an  0Tnc{<>>«6T0iMwd 


Infrared  Seeker  Performance  Metrics 


A02-158:  Phase  I SBIR 


Final  Report 


December  31, 2003 


DISTRIBUTION  STATEMENT  A 
Approved  for  Public  Release 
Distribution  Unlimited 


Invariant  Corporation 


2^209  088 


REPORT  DOCUMENTATION  PAGE 

Form  Approved 

0MB  No.  0704-0188 

Public  reporting  burden  for  this  collection  of  information  Is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources, 
gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this 
collection  of  information,  including  suggestions  for  r^ucing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson 
Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302,  and  the  Office  of  Management  and  Budget,  paperwork  Reduction  Project  (0704-0188),  Washington,  DC  20503. 

1.  AGENCY  USE  ONLY  (Leave 

Blank) 

2.  REPORT  DATE 

31  Dec  2003 

3.  REPORT  TYPE  AND  DATES  COVERED 

Final:  Jan  2003  -  Dec  2003 

4.  TITLE  AND  SUBTITLE 

Infrared  Seeker  Performance  Metrics 

5.  FUNDING  NUMBERS 

DAAH01-03-C-R129 

6.  AUTHOR(S) 

David  R.  Anderson,  Jim  Moore,  John 

Montgomery,  and  Mark  Chambliss 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Invariant  Corporation  Dynetics 

4800  Whitesburg  Dr  #30-353  1 000  Explorer  Boulevard 

Huntsville,  AL  35802  Huntsville,  AL  35806 

8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 

INV-TR-03-001 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

US  Army  Aviation  and  Missile  Command 

ATTN:  AMSAM-RD-MG-IR 

Redstone  Arsenal,  AL  35898 

Lisa  B.  Cannon 

10.  SPONSORING/MONITORING 
AGENCY  REPORT  NUMBER 

11.  SUPPLEMENTARY  NOTES 

12a.  DISTRIBUTION/AVAILABILITY  STATEMENT  (see  Section  5.3b  of  this  solicitation) 

Approved  for  public  release;  distribution  unlimited 

12b.  DISTRIBUTION  CODE 

1 3.  ABSTRACT  (Maximum  200  words) 

Report  developed  under  SBIR  contract  for  topic  A02-158.  Advances  in  imaging  infrared  (HR)  technology  and  demonstration 
of  this  technology  as  a  capable  means  of  target  discrimination,  automatic  target  recognition  (ATR),  and  auto-tracking  have  led 
to  the  development  of  numerous  HR  weapon  systems.  Although  excellent  analysis  tools  exist  for  describing  the  imaging 
sensors  themselves,  no  adequate  method  or  tools  exist  for  characterizing  the  auto-detection  and  tracking  performance 
capability  of  the  sensors  against  targets  in  a  variety  of  backgroimds.  This  is  complicated  by  the  fact  that  auto-detection  and 
tracking  techniques  are  difficult  to  characterize.  It  is  impossible  to  generate  a  single  generic  metric  that  will  accurately  predict 
the  performance  of  all  imaging  auto-trackers.  Typically  auto-trackers  can  be  categorized  based  on  their  fundamental 
algorithm.  With  knowledge  of  the  detection  or  tracking  algorithm,  an  appropriate  metric  can  be  used  to  predict  performance. 
This  effort  identifies  the  common  detection  algorithms  and  tracker  routines  and  uses  the  fundamental  algorithms  as  metrics. 
These  metrics  will  be  used  to  analyze  real  imagery  from  various  IR  sensors.  A  methodology  for  a  performance  metric  will  be 
developed  that  accurately  predicts  auto-detection  and  tracker  performance  and  a  validation  plan  will  be  developed  comparing 
actual  auto-detection  and  tracker  systems  to  the  metric  results. 

As  the  U.S.  Army  moves  forward  in  its  use  of  IIR  technology,  development  of  a  tool  capable  of  predicting  auto-detection  and 
tracker  performance  is  essential  for  optimizing  algorithm  development  and  setting  seeker  system  parameters. 

14.  SUBJECT  TERMS  ~  15.  NUMBER  OF  PAGES 

SBIR  Report,  Signature  Metrics,  Seeker  Performance,  Auto-tracker  Performance,  ATD  Performance  1 05 


17.  SECURITY  CLASSIFICTION  OF  18.  SECURITY  CLASSIFICATION  OF  19.  SECURITY  CLASSIFICATION  OF  I  20.  LIMITATION  OF  ABSTRACT 


REPORT 


Unclassified 


THIS  PAGE 


Unclassified 


ABSTRACT 


Unclassified 


UL 


REFE 


Report  Number:  INV-TR-03-001 


SBIR  A02-158  Phase  I:  Infrared  Seeker  Performance  Metrics 

Final  Report 


Prepared  for: 

US  Army  Aviation  and  Missile  Command 
ATTN:  AMSAM-RD-MG-IR 


Redstone  Arsenal,  AL  35898 
Contract  Number :  DAAH01-03-C-R129 
CDRL  AOOl 


Government  Technical  POC 

Lisa  B.  Cannon 


Invariant  Technical  POC 
David  R.  Anderson 
Invariant  Corporation 
4800  Whitesburg  Drive  #30-353 
Huntsville,  AL  35802 
Phone:  (256)  885-9794 


December  31, 2003 


4 


REPRODUCED  FROM 
BEST  AVAILABLE  COPY 


ABSTRACT 


This  final  report  summarizes  the  activities  of  Invariant  Corporation  in  its  support 
of  the  Army  Aviation  and  Missile  Command  (AMCOM)  under  contract  DAAH01-03-C- 
R129.  The  effort  was  a  Phase  I  SBIR  entitled  A02-158,  Infrared  Seeker  Peffomtance 
Metrics.  The  report  details  the  technical  effort  performed  to  include  identification  of 
infrared  sequences  and  the  ground-truthing  of  these  sequences.  Signature  metrics  were 
identified  and  developed  to  process  statistical  differences  between  target  and  clutter. 
Software  was  developed  to  execute  the  metrics  and  was  a  deliverable  under  this  effort.  A 
tracker  and  detection  prediction  methodology  study  was  identified  and  a  validation  plan 
for  this  methodology  is  detailed.  This  effort  was  also  supported  by  Dynetics  which  was  a 
subcontractor  to  Invariant  Corporation  on  this  effort. 


APPROVED: 

David  R.  Anderson 
President 


5 


TABLE  OF  CONTENTS 


L  INTRODUCTION . . . 9 

2.  TECHNICAL  OBJECTIVES . 9 

X  TECHNICAL  WORK . 11 

3.1  Infrared  Image  Sequences . . 11 

3.2  Metrics . 16 

3.2.1  EXISTING  Metrics . ....16 

3.2.2  New  Metrics . 18 

3.2.3  Tracker  Performance  Metric . 33 

3.3  Signature  Metric  Software  Tool . 36 

3.4  Ground  Truth  Process . ..41 

3.5  Methodology  for  Seeker  Performance  Predictions . 42 

3.5. 1  Performance  Prediction  Methodology  Problem . . . 42 

3.5.2  Performance  Prediction  Solution . 43 

3.5.3  Development  of  Basic  Neurai.  Network  Approach . 44 

3.6  Validation  Plan . 51 

3.6.1  MDTP  Metric  Tool . 51 

3.6.2  MDTP  Probability  oi  Detfxtion  Pri;diction  Methodology . 52 

3.6.3  MDTP  Probability  oi  Track  Prediction  Methodology . 52 

3.6.4  MDTP  MODi;i.  VALIDATION . 55 

3.6.5  Validation  Criti;ria . 58 

3.6.6  Statisticai.  Analysis . 59 

4.  TRACKER  ALGORITHM . 69 


5.  CONCLUSIONS . 71 


6 


LIST  OF  FIGURES 


Figure  1.  Phase  I  Block  Diagram . ....10 

Figure  2.  Ambient  Temperature  Histogram . 12 

Figure  3.  Relative  Humidity  Histogram . . . 13 

Figure  4.  Image  Sequence  Time-of-Day . 14 

Figure  5.  Image  Sequence  Season . 15 

Figure  6.  Correlation  Area . . . 20 

Figure  7.  High  Contrast  Target . 22 

Figure  8  Correlation  Surface  of  High  Contrast  Target . 22 

Figure  9.  Low  Contrast  Target . 23 

Figure  10.  Correlation  Surface  of  Low  Contrast  Target . 24 

Figure  11.  Track  Correlation  Metric  Output  for  High  Contrast  Target . 25 

Figure  12.  Track  Correlation  Output  for  Low  Contrast  Target . 26 

Figure  13.  Sobel  Filter  Results . 28 

Figure  14.  Image  With  Target  and  Background  Gates  Superimposed . 30 

Figure  15.  Tracker  Performance  Metric . 34 

Figure  16.  Track  Correlation  Metric  Comparison  to  Tracker  Performance . 35 

Figure  17.  Image  Metric  GUI . 37 

Figure  18.  UML  Object  Interaction . 41 

Figure  19.  TAGtool  Screen  Capture . 42 

Figure  20.  Linear  Spatial  Filter . 44 


7 


Figure  21.  General  Multi-Layer  Perceptron  Architecture . 46 

Figure  22.  Signal-Flow  Highlights  of  Output  Neuron  j . 47 

Figure  23.  Essential  Steps  for  Validation  of  Models  and  Simulations . 54 

Figure  24.  Validation  Methodology . 55 

Figure  25.  Graphical  Comparison  with  Confidence  Intervals . i . 59 

Figure  26.  Validation  Results  and  Types  of  Errors . 61 

Figure  27.  Cl  Width  versus  Number  of  Test . 64 

Figure  28.  Cl  Width  versus  Pd . 65 

Figure  29.  Effect  of  Number  of  Tests  on  OC  Curve . 68 


8 


1.  Introduction 


Advances  in  imaging  infrared  (HR)  technology  and  demonstrations  of  this 
technology  as  a  capable  means  of  target  discrimination,  automatic  target  recognition 
(ATR),  and  auto-tracking  have  led  to  the  development  of  numerous  HR  weapon  systems. 
No  doubt,  as  the  technology  continues  to  improve,  additional  Department  of  Defense 
(DoD)  time  and  resources  will  be  spent  in  an  effort  to  improve  the  detection, 
classification,  and  guidance  capabilities  of  US  assets.  Although  excellent  analysis  tools 
exist  for  describing  the  imaging  sensors  themselves,  there  are  no  adequate  methods  or 
tools  currently  in  existence  for  characterizing  the  performance  capability  of  the  sensors 
against  targets  in  a  variety  of  backgrounds.  Thus,  new  and  improved  detection  and 
tracker  algorithms  continue  to  be  developed,  but  with  no  technique  for  predicting  their 
potential  performance  enhancement. 

Performance  metrics  and  related  analysis  tools  have  been  developed  for  man-in- 
the-loop  applications  that  adequately  match  predicted  performance  with  human 
perception  test  results.  While  similar  metrics  have  been  developed  based  upon  auto¬ 
detection  and  tracker  test  results,  a  reliable  method  using  these  metrics  in  predicting  the 
performance  of  trackers  and  auto-detection  algorithms  for  a  variety  of  targets  in  diverse 
backgrounds  has  not  been  realized.  As  the  US  Army  moves  forward  in  its  use  of  HR 
technology,  development  of  a  tool  capable  of  predicting  sensor  performance  is  essential 
for  optimizing  algorithm  development  and  seeker  system  design.  From  a  defensive 
standpoint,  as  foreign  armies  implement  HR  capabilities  into  their  weapon  systems,  such 
a  tool  is  also  necessary  in  mitigating  risk  to  US  ground  vehicles  and  troops. 

2.  Technical  Objectives 

The  overall  objective  of  this  effort  was  to  investigate  and  develop  metrics  and 
methodologies  which  can  be  used  to  predict  the  auto-detection  and  tracking  performance 
of  imaging  infrared  missile  seekers  that  employ  staring  focal  plane  arrays,  and  develop  a 
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plan  to  validate  the  performance  metrics.  The  specific  objectives  of  the  proposed  effort 
are  in  Figure  1  and  are  listed  below. 


1.  Identify  a  set  of  infrared  image  sequences  that  represent  a  variety  of  background 
conditions,  sensor  resolution,  and  sensor  sensitivity. 

2.  Identify  existing  signature  metrics  and  formulate  new  ones. 

3.  Develop  a  software  tool  to  use  for  calculating  signature  metrics. 

4.  Ground-truth  the  image  sequences  identified  in  objective  1  and  calculate  the 
signature  metrics  for  each  image  sequence. 

5.  Develop  a  methodology  for  predicting  auto-detection  and  tracker  performance 
based  on  the  signature  metrics. 

6.  Develop  a  plan  to  validate  the  performance  metrics. 


Figure  1.  Phase  I  Block  Diagram 
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3.  Technical  Work 


3.1  Infrared  Image  Sequences 

The  first  portion  of  this  effort  was  to  identify  a  comprehensive  set  of  infrared 
image  sequences  for  metric  processing  and  then  later  in  the  Phase  II  to  support  the  tracker 
and  Autonomous  Target  Detection  (ATD)  prediction  tool  validation.  The  goal  was  to 
identify  sequences  with  various  backgrounds,  sensor  sensitivity,  and  resolution  to  ensure 
the  future  analysis  results  were  not  biased  to  just  one  sensor  type  or  a  single 
environmental  condition.  Parameters  that  cause  variations  in  background,  sensor 
resolution,  and  sensitivity  were  identified  and  this  information  was  recorded  for  each 
image  sequence  selected.  In  all,  714  image  sequences  were  identified.  The  sequences 
are  collected  from  both  tower  test  and  captive  flight  tests  (CFT).  Each  sequence  contains 
anjwhere  from  300  to  8000  images.  Most  tower  test  sequences  have  targets  at  a  constant 
range,  while  the  CFT  sequences  begin  at  an  initial  range  and  then  close  on  the  target. 
The  sequences  were  selected  from  several  sensors  that  have  different  resolution  and 
sensitivity.  First,  let’s  summarize  the  weather  conditions  for  the  images  sequences 
selected. 

For  each  sequence,  various  weather  conditions  were  recorded  while  the  data  was 
collected.  These  parameters  varied  somewhat  depending  on  the  location  of  the  test  and 
the  instrumentation  that  was  available.  The  main  information  recorded  was  location, 
time-of-day,  season,  ambient  air  temperature,  relative  humidity,  parametric  pressure, 
wind  speed,  wind  direction,  dew  point,  and  precipitation.  At  some  sites,  soil  temperature, 
visibility,  and  solar  radiation  were  also  recorded.  All  weather  related  parameters  are 
stored  in  a  database  that  is  a  deliverable  of  this  Phase  I.  To  show  variability,  some  of 
these  parameters  are  plotted  below.  Figure  2  is  a  histogram  of  the  ambient  air 
temperature  of  all  of  the  sequences.  There  is  a  large  concentration  around  75  to  80 
degrees  F,  but  several  of  the  sequences  were  around  90  deg  F  and  during  some  of  the 
winter  scenarios,  there  were  some  temperatures  in  the  upper  30’s. 
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Figure  2.  Ambient  Temperature  Histogram 

Figure  3  is  a  histogram  of  the  relative  humidity  of  the  selected  sequences.  There 
is  a  concentration  of  sequences  with  high  relative  humidity  but  there  is  also  a  large  group 
of  sequences  varying  between  30  and  100  percent. 
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Figure  3.  Relative  Humidity  Histogram 

One  of  the  most  important  aspects  of  an  image  sequence  is  the  time-of-day  in  which  the 
data  was  collected.  Target  to  background  signature  will  obviously  vary  greatly  from  the 
middle  of  the  night  to  the  heat  of  the  day.  Figure  4  shows  the  time  in  which  the 
sequences  selected  were  generated.  Typically,  most  data  is  collected  during  the  day, 
either  morning  or  afternoon.  But  there  is  a  decent  percentage  of  this  data  that  was 
collected  at  night. 
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Figure  4.  Image  Sequence  Time-of-Day 

The  final  illustration  is  the  season  in  which  the  data  sequences  were  collected.  Figure  5 
identifies  the  number  of  runs  collected  during  the  different  seasons.  Unfortunately,  there 
were  no  sequences  collected  during  the  summer  months  in  the  set  that  was  identified. 
Nonetheless,  several  of  the  spring  sequences  were  late  in  the  season  and  had  very  hot 
summer-like  conditions. 

This  section  lists  just  a  subset  of  the  weather  conditions  recorded  during  data 
collection.  Additional  weather  parameters  were  stored  and  are  available  for  each 
sequence  in  Appendix  A. 
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Figure  5.  Image  Sequence  Season 

Sensor  sensitivity  was  the  next  important  parameter  to  be  considered  when 
identifying  the  infrared  sequences.  This  is  usually  measured  by  examining  sensor  fixed 
pattern  noise  and  temporal  noise  or  Noise  Equivalent  Delta  Temperature  (NEdT).  For 
various  reasons,  these  numbers  are  not  always  publicized.  The  sequences  identified  in 
this  effort  were  generated  by  4  separate  sensors.  The  database  delivered  with  this  report 
identifies  the  sensors  and  gives  a  sensitivity  performance  range.  NEdT  for  the  different 
sensors  varies  from  just  below  50  mK  to  greater  than  100  mK. 

Finally,  sensor  resolution  is  an  important  consideration  that  provides  variation  in 
the  image  sequences.  Resolution  includes  spatial  pixel  resolution,  spectral  resolution, 
optical  resolution  (blur)  and  grayscale  resolution.  The  sensors  used  varied  from  512  x 
512,  384  X  512,  and  256  x  256.  Grayscale  resolution  was  both  12  bits  and  14  bits  per 
pixel.  The  instantaneous  FOV  was  different  for  each  of  the  sensors  as  was  the  optical 
blur.  All  image  sequences  were  collected  using  midwave  infrared  sensors.  There  are 
currently  two  requests  for  long  wave  infrared  data,  but  at  the  completion  of  the  Phase  I, 
no  LWIR  starring  focal  plane  array  data  had  been  obtained.  But  even  though  each  of  the 
sequences  were  operating  in  the  midwave  band,  they  all  operated  in  a  slightly  different 
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sub-region  of  the  3  to  5  band.  The  variation  in  spectral  band  will  generate  variation  in  the 
resulting  sequences. 

All  sequences  in  Appendix  A  have  been  ground-truthed.  The  proposed  Phase  1 
option  is  to  obtain  and  ground-truth  additional  infrared  sequences.  Due  to  pending  CFT 
data  collect,  good  long  wave  infrared  (LWIR)  sequences  should  be  available  by  that  time. 

3.2  Metrics 

3.2.1  Existing  Metrics 

The  existing  metrics  identified  are  traditional  first  order  metrics  that  compare 
basic  statistics  between  the  target  area  and  the  background  around  the  target.  The  three 
classical  metrics  are  ATrss.  ATmodified.  and  SCRrss-  The  standard  statistics  used  by  all 
three  are  mean  temperature  of  the  target  area  and  background,  and  the  standard  deviation 
in  a  specific  area.  The  metrics  compare  the  statistics  calculated  in  a  target  region  to  the 
statistics  calculated  in  the  background  area.  The  similarities  of  these  statistics  are  the 
basis  for  the  metrics.  The  classical  metrics  all  define  the  target  area  and  the  background 
area  the  same.  The  target  area  in  which  the  target  statistics  are  calculated  is  defined  as 
the  bounded  box  around  the  target.  It  is  identified  during  the  ground-truth  process. 
Ground-truthing  will  be  discussed  in  detail  later  in  this  report.  An  area  around  the  target 
gate  is  evaluated  for  background  statistics.  Typically,  the  background  area  is  the  ratio  of 
3  times  the  target  height  and  2.5  times  the  target  width.  The  classical  metrics  are 
described  in  the  following  sections. 


3.2.1. 1  Delta  Temperature  Root  Sum  Squared  (ATrss) 

This  metric  takes  the  delta  temperature  between  the  mean  pixels  in  the  target  area 
and  the  mean  pixels  in  the  background  area.  It  also  calculates  the  standard  deviation  of 
the  pixels  in  the  target  area  (or).  These  two  terms  are  individually  squared,  summed  and 
then  the  square  root  of  the  sum  formulates  the  ATrss  metric  in  Equation  1. 
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Equation  1 


AT,ss=Jim^ 


3.2.1. 2  Delta  Temperature  Modified  (ATmodiHed) 

This  metric  is  very  similar  to  the  ATrss  metric.  It  calculates  target  and 
background  statistics  in  the  same  regions  of  interest.  The  only  difference  is  background 
clutter  standard  deviation  is  also  calculated  in  the  background  area.  It  is  subtracted  from 
the  target  sigma  before  being  squared  and  then  added  to  the  square  of  the  delta 
temperature  term.  The  addition  of  background  clutter  sigma  is  very  important  since 
background  clutter  will  have  a  significant  impact  on  tracker  and  ATA  performance. 
Equation  2  describes  the  ATmodified  calculation 

A7:,w».  =  i/<An=  +  ((Tr-<^c)’  Equation  2 


3.2.1. 3  Signal  to  Clutter  Ratio  (SCRrss) 

This  metric  has  a  strong  dependence  on  the  sigma  of  the  clutter  around  the  target. 
It  is  the  classical  signal  to  clutter  ratio,  but  uses  ATrss  for  the  signal  term  instead  of 
simply  AT.  This  is  important  since  even  in  the  absence  of  target  mean  difference  from 
the  background,  target  sigma  alone  is  a  significant  contributor  to  the  ability  to  track  or 
detect  targets.  Equation  3  describes  the  SCRrss  calculation. 


(Tf'D  _  ^^RSS 


Equation  3 


Classical  metrics  have  been  used  for  years,  but  to  date  still  do  not  do  an  adequate 
job  of  predicting  tracker  performance.  Trackers  respond  to  spatial  structure  and 
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similarities  between  the  target  and  the  background.  These  metrics  do  not  allow  for 
spatial  frequency  contribution  and  are  purely  first  order  statistics.  They  also  process  on  a 
single  frame  of  data.  This  is  inconsistent  with  the  auto-tracker  process  of  using  multiple 
frames  of  data  to  generate  a  history  of  the  target  information.  Unless  otherwise 
incorporated  into  time  average  filter  or  other  custom  process,  temporal  information  is 
negated  with  these  metrics. 

3.2.2  New  Metrics 

The  signal  to  clutter  metrics  described  above  are  effective  for  a  low  level 
estimation  of  IR  seeker  performance.  But  often  these  metrics  are  not  a  good  measure  to 
describe  the  impact  of  the  target-background  signature  has  on  the  tracking  process.  More 
complicated  image  metrics  are  required  to  support  this  analysis.  Tracker  and  ATD 
algorithm  metrics  can  be  developed  that  match  more  closely  the  image  processing  that  is 
performed  by  the  trackers  being  evaluated.  Most  imaging  auto-tracker  and  ATD 
algorithms  implemented  in  systems  today  are  either  company  proprietary  or  classified. 
For  this  reason,  no  specific  algorithms  will  be  described.  Instead,  generalizations  can  be 
made  with  the  knowledge  of  existing  tracker  and  ATD  implementations  in  open 
literature. 

Trackers  are  typically  categorized  based  on  the  type  of  image  processing 
performed.  They  all  have  one  thing  in  common.  They  attempt  to  maintain  track  gates 
around  the  target  of  interest,  and  they  attempt  to  maintain  an  aimpoint  on  some  portion  of 
the  target.  The  first  type  of  tracker  is  a  hot  spot  tracker.  This  is  the  simplest  tracker  type. 
The  algorithm  processes  the  image,  and  finds  the  hottest  pixel  intensity.  In  some  cases, 
the  image  is  pre-processed  with  a  boxcar  averaging  filter,  to  eliminate  noise.  But  in 
either  case,  the  hottest  pixel  in  the  scene  or  processed  scene  is  classified  as  target.  This 
pixel  is  tracked  and  the  aimpoint  is  placed  on  the  hot  spot. 

The  second  type  tracker  is  statistical  based.  Target  and  background  means  and 
sigmas  are  calculated.  The  tracker  will  evaluate  the  similarities  between  the  statistics  and 
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then  classify  pixels  in  the  scene  as  target  and  background  based  on  the  calculation. 
Bayes’  law  is  often  used  to  perform  the  classification. 

The  third  type  of  tracker  is  a  feature-based  tracker.  Typically,  the  image  is  pre¬ 
filtered  using  some  technique.  There  are  several  filters  to  choice  from,  but  the  Sobel 
edge  enhancement  filter  seems  to  be  a  popular  choice.  It  does  a  very  good  job  of 
enhancing  edges  in  the  image,  and  can  be  implemented  using  2  3x3  convolution  masks. 
This  makes  real-time  execution  possible  in  a  wide  variety  of  hardware  platforms.  Using 
the  processed  image,  features  are  extracted  using  some  segmentation  criteria.  A  database 
of  features  is  generated  and  maintained  that  describes  characteristics  of  each  of  the 
features.  This  often  includes  feature  position,  magnitude,  direction,  size  and  velocity. 
The  features  are  classified  as  target  or  background  based  on  criteria  that  vary  from  tracker 
to  tracker.  These  features  are  used  to  determine  the  track  gate  size  and  aimpoint. 

The  final  tracker  is  a  correlation-based  tracker.  This  type  of  tracker  uses  target 
template  information,  often  generated  by  the  gunner  at  lock-on,  and  maintains  the  track 
gate  on  the  target.  A  correlation  technique  between  the  target  template  and  the  sensor 
image  is  performed  to  calculate  the  offset  of  the  target  in  the  scene  from  its  previous 
location.  There  are  several  methods  of  performing  the  image  correlation  but  given  the 
same  template  and  image  correlation  area,  they  will  all  generate  similar  results. 

ATD  algorithms  typically  use  a  target  template  and  search  the  entire  image  to 
identify  features  that  match  the  template.  The  image  is  often  preprocessed  to  extract 
edges  or  high  frequency  components.  Many  ATD  algorithms  use  a  process  similar  to  the 
correlation-based  tracker,  but  search  for  a  good  correlation  in  the  entire  image  instead  of 
a  local  region  around  the  previous  track  location. 

The  goal  of  this  effort  was  to  identify  at  least  one  new  metric  that  closely  matches 
the  fundamental  track  and  ATD  algorithms  described  above. 
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3.2.2.1  Track  Correlation  Metric 


The  track  correlation  metric  (TCM)  correlates  between  the  current  target  template 
containing  previous  target  information  and  a  correlation  search  region.  The  search  region 
is  defined  by  a  box  around  the  target  plus  a  correlation  search  area.  Figure  6  shows  an 
example  of  the  search  area  outlined  with  the  green  box.  The  blue  box  outlines  the  area 
that  defines  the  target  template. 


Figure  6.  Correlation  Area 

The  target  template  is  a  2  dimensional  image  that  has  j  rows  and  i  columns.  The 
correlation  search  area  has  n  rows  and  m  cols,  n  is  3  times  the  size  of  j  and  i  is  3  times 
the  size  of  m.  The  first  step  in  this  metric  is  to  perform  a  normalized  correlation  between 
the  correlation  search  area  from  (n,  m)  to  (n+j,  m+i)  where  n  and  m  are  initially  zero  and 
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the  target  template.  To  normalize  the  output,  the  mean  and  sigma  value  must  first  be 
calculated  for  the  correlation  search  area.  The  mean  is  defined  as  pc  and  the  sigma  is  Oe. 
Then  the  mean  and  sigma  is  calculated  for  the  target  template.  These  are  defined  as  pt 
and  Ot  respectively.  The  normalized  cross  correlation  is  defined  by  Equation  4.  The  result 
is  output  to  a  matrix  called  the  correlation  surface.  This  surface  is  populated  with  results 
as  you  change  the  starting  points  n  and  m  to  change  the  location  of  the  correlation  in  the 
correlation  search  area. 


Equation  4 


Below  are  two  examples  of  the  resulting  correlation  surface.  Figure  7  is  the 
output  of  an  InSb  MWIR  sensor.  The  target  is  hot  with  respect  to  the  background  and  is 
outlined  by  the  blue  box.  When  performing  a  normalized  cross-correlation  with  this 
input  the  result  should  be  a  correlation  surface  with  a  sharp  peak.  The  target  area  is  not 
well  correlated  with  surrounding  clutter,  so  the  correlation  surface  values  should  be  small 
off  the  correlation  peak.  Figure  8  shows  the  correlation  surface  result  after  processing  the 
high  contrast  image.  As  expected,  values  around  the  peak  in  the  surface  quickly  go  to 
zero,  and  there  are  no  secondary  peaks  anywhere  in  the  surface. 
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The  next  example  is  a  low  contrast  target  on  somewhat  correlated  clutter.  Figure 
9  shows  the  IR  image  with  the  target  outlined  in  blue.  After  performing  the  normalized 
cross-correlation.  Figure  10  shows  the  resulting  correlation  surface.  As  expected,  the 
surface  has  the  peak  in  the  center  resulting  from  the  correlation  between  the  target 
template,  and  the  target  itself.  But  the  slope  from  the  peak  is  less  than  the  previous 
example,  and  then  the  values  ramp  back  up  away  from  the  peak.  This  indicates  clutter 
that  is  correlated  to  the  target. 
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Figure  10.  Correlation  Surface  of  Low  Contrast  Target 

The  trick  now  is  to  use  the  information  from  the  correlation  surface  to  generate  a 
metric  value  that  is  the  result  of  the  target  being  correlated  to  the  background.  Since  the 
surface  is  normalized,  you  will  always  have  a  peak  value  of  the  target  template 
correlating  with  the  target  in  the  correlation  search  area.  There  are  also  values 
immediately  around  the  peak  that  are  a  result  of  the  target  correlating  with  itself.  Since 
the  real  interest  is  to  determine  how  alike  the  target  is  to  background,  the  values  of 
interest  are  where  the  clutter  correlated  to  the  target  template.  Remember  that  the  target 
template  is  a  historical  snapshot  of  the  actual  target,  hence  the  perfect  correlation. 

So  to  generate  the  metric  value,  the  correlation  surface  away  form  the  peak  is 
examined.  If  there  is  a  value  anywhere  in  the  correlation  surface  that  has  a  large  value 
that  means  it  is  highly  correlated  with  the  background.  Lower  values  in  the  correlation 
surface  indicate  poor  correlation.  This  is  opposite  from  what  is  expected  from  a  single 
metric  value,  since  a  1  should  indicate  a  high  likelihood  of  distinguishing  target  from 
background,  and  a  zero  indicates  the  target  and  background  are  highly  correlated. 
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Therefore,  the  largest  value  in  the  correlation  surface  that  is  outside  the  peak  region  is 
subtracted  from  1 .0.  The  result  is  the  final  metric  value. 


Track  Correlation  Metric  (TCM)  =  1.0  -  correlation  peak(ouKidc  center  region)  Equation  5 


Figure  1 1  is  an  example  of  the  TCM  output.  The  target  is  high  contrast  just  off  a 
road  as  indicated  by  the  red  arrow.  The  metric  value  is  around  0.8  for  the  entire  sequence 
indicating  there  is  a  high  probability  of  a  successful  track  on  this  image  sequence. 
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Figure  11.  Track  Correlation  Metric  Output  for  High  Contrast  Target 

Figure  12  is  another  example  of  the  TCM  output.  The  target  is  low  contrast  and 
identified  by  the  red  arrow.  The  metric  value  is  around  0.4  to  0.6  for  the  entire  sequence 
indicating  there  is  a  medium  to  low  probability  of  a  successful  track  on  this  image 
sequence. 
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Figure  12.  Track  Correlation  Output  for  Low  Contrast  Target 

The  tracker  correlation  metric  uses  a  target  reference  that  is  filtered  over  time. 
Since  auto-trackers  react  to  temporal  changes  in  the  target  area,  it  was  important  that  the 
metric  is  sensitive  to  temporal  changes  as  well.  This  is  true  because  they  use  historical 
target  and  background  information  for  discrimination  and  classification.  For  this  analysis 
of  the  tracker  correlation  metric,  the  target  template  was  a  stored  representation  of  the 
target  area  from  previous  frames.  If  there  is  a  significant  change  in  signature  from  one 
frame  to  the  next,  the  metric  should  perform  well  when  predicting  the  effect  on  tracker 
performance.  Since  this  metric  uses  the  fundamental  algorithms  used  by  a  correlation- 
based  tracker,  the  metric  value  should  do  a  fairly  good  job  of  predicting  the  performance 
at  that  class  of  tracker. 
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3.2.2.2  Sobel  Metric 


Another  metric  under  consideration  for  track  analysis  is  based  on  the  Sobel  edge 
enhancement  mask*.  It  is  used  to  evaluate  the  ability  to  pull  edges  out  of  a  particular 
scene.  Equation  6  represents  the.  horizontal  Sobel  mask  and  Equation  7  is  the  vertical 
mask  applied  to  the  raw  input  image  in  an  area  around  the  target  gate.  These  images  are 
combined  to  form  a  magnitude  image  calculated  in  Equation  8.  Currently,  the  target  area 
of  the  edge  image  is  compared  to  a  background  area  to  generate  a  signal  to  clutter  ratio. 
This  ratio  is  an  indication  of  how  well  the  filtered  edges  on  the  target  compare  to  the  edge 
features  in  the  background  clutter  around  the  target.  Future  work  will  include  a  more 
exhaustive  analysis  of  the  persistence  of  the  edge  information.  This  will  form  the  ability 
to  predict  how  well  a  feature  based  tracker  can  maintain  consistent  edge  features  over  the 
duration  of  track.  The  filtered  images  in  Figure  13  illustrate  the  edge  enhancement 
effects  of  the  Sobel  mask. 


-1  -2  -1 
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1  2  1 

Horizontal  Sobel  Mask 


Equation  6 


-1  0  1 
-2  0  2 
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Vertical  Sobel  Mask 


Equation  7 


magnitude^  =  yjGx+Gy 


Equation  8 
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Horizontal  Edge 


Vertical  E^e 


Figure  13.  Sobel  Filter  Results 

3.2.2.3  Bayesian  Based  Signature  Metric 

The  Bayesian  metric  describes  the  separability  of  the  target  and  background  based 
on  their  statistical  signatures  (i.e.  it  is  a  relative  metric).  The  metric  assumes  a  grotmd- 
truth  process  has  identified  the  target.  The  output  of  the  metric  is  a  value  between  0  and 
1.  Values  near  1  represent  targets  that  are  very  separable  from  the  background  and  values 
near  0  represent  targets  that  are  very  similar  to  the  background. 

The  Bayesian  metric  classifies  pixels  based  on  their  similarity  to  statistical  models 
for  the  target  and  background.  The  metric  gate  (MG)  of  the  Bayesian  metric  has  two 
components,  the  target  pixel  gate  (TPG)  and  the  background  pixel  gate  (BPG).  The  TPG 
is  centered  on  the  target  and  matched  to  the  size  of  the  target.  The  BPG  is  also  centered 
on  the  target  but  has  larger  than  the  size  of  the  TPG  (typical  value  of  3  times  the  TPG 
dimensions).  The  BPG  excludes  the  area  designated  by  the  TPG.  The  target  statistical 
model  is  formed  based  on  the  pixels  inside  the  TPG.  The  background  model  uses  only 
pixels  inside  the  BPG.  The  mean,  standard  deviation,  and  correlation  coefficient  are 
calculated  for  the  TPG  and  BPG  to  determine  the  bivariate  normal  distributions  for  the 
target  and  background  pixels.  Calculation  of  the  correlation  coefficient  requires  the  user 
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to  specify  the  offsets  between  the  current  pixel  and  its  statistical  pair.  This  offset  is 
configurable  (typical  values  are  an  offset  of  1  in  both  the  horizontal  and  vertical 
directions).  These  statistics  can  be  established  on  a  single  frame  or  recursively  updated 
over  several  frames  to  provide  a  means  of  memory  and  adaptation  in  the  statistical  model. 
Once  the  statistical  models  are  established,  each  pixel  within  the  MG  can  be  classified 
according  to  its  probability  of  belonging  to  either  the  background  or  target  class  based  on 
Bayes’  law.  The  metric  is  then  calculated  by  the  average  success  of  correctly  classifying 
target  pixels  as  belonging  to  the  target  class  and  background  pixels  as  belonging  to  the 
background  class. 

The  Bayesian  metric  classifies  pixels  within  a  region  as  belonging  to  one  of  two 
groups,  target  or  background,  based  on  Bayes’  law.  The  two  groups  are  assumed  to 
follow  bivariate  normal  distributions  with  characteristic  parameters  being  the  mean, 
standard  deviation,  and  correlation  coefficient. 

The  MG  is  centered  on  the  designated  target  position  via  the  ground  truth 
information  and  the  mean,  standard  deviation,  and  correlation  coefficient  are. calculated 
for  both  the  target  and  the  background  ^eas.  On  the  cuirent  or  subsequent  frames  (user 
configurable),  the  likelihood  of  each  pixel  belonging  to  either  the  target  or  background 
class  is  calculated. 

The  TPG  and  BPG  are  each  assumed  to  have  pixel  pairs  that  can  be  described 
with  the  bivariate  normal  distribution  function.  A  spatial  relationship  defined  by  the 
horizontal  and  vertical  offsets  (XOFF  and  YOFF)  is  used  to  pair  pixels  together.  The 
same  offsets  are  used  for  the  target  and  the  background  areas.  It  is  important  that  the 
offsets  not  exceed  one-half  the  target  size  so  the  majority  of  target  pixels  are  paired  with 
other  target  pixels.  XOFF  and  YOFF  are  parameters  specified  by  the  user. 

To  determine  the  target  and  background  statistical  parameters,  rectangular  gates 
TPG  and  BPG  are  used  (Figure  14).  The  user  specifies  the  size  and  initial  location  of  the 
TPG  and  BPG.  The  centers  of  the  two  gates  are  located  at  the  same  position  in  the  image. 
The  pixels  in  the  TPG  are  excluded  from  the  BPG  calculations;  therefore,  the  BPG  must 
be  larger  than  the  TPG.  Within  the  gates,  pixel  pairs  are  formed  and  the  statistics 
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computed.  The  statistics  calculated  are  the  mean  pixel  value,  standard  deviation,  and 
correlation  coefficient. 


Figure  14.  Image  With  Target  and  Bacl^round  Gates  Superimposed 

Each  pixel  is  classified  by  determining  its  target  likelihood,  L, ,  in  terms  of  the 
prior  probabilities  for  the  target  and  background,  p{g,  )  and  /’(G^ ) ,  and  the  conditional 
probability  distribution  functions  for  the  target  and  background,  ^(vjVjlG,)  and 
F(vjV2  I  Gj,  ) ,  as  shown  in  Equation  9.  The  Vi  term  is  the  value  of  the  current  pixel  being 
evaluated.  The  vj  term  is  the  value  of  the  statistical  pair  to  vj  located  XOFF  and  YOFF 
from  the  location  of  vj .  The  G,  term  is  the  group  of  target  pixels  designated  by  the  TPG 
and  Gfc  is  the  background  pixels  inside  the  BPG  excluding  the  TPG. 


L,  =P(G,  lviV2)  = 


P{v,v,\G,)p{G,) 

P(v,V2|G,)p(G,)  +  />(v,V2|G,MGj 


Equation  9 
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The  prior  target  probability  P{G, )  is  the  ratio  of  the  number  of  target  pixels  to  the 
total  number  of  pixels  within  the  MG  as  shown  in  Equation  10.  The  background  term 
p{Gi,  )  is  the  ratio  of  the  number  of  background  pixels  (MG-TPG),  to  the  total  number  of 
pixels  in  the  MG  as  shown  in  Equation  1 1 .  The  values  of  P{G, )  and  P{G^ )  represent  the 
probabilities  of  a  pixel  belonging  to  the  target  or  background  group  without  knowledge  of 
its  value. 


P{G,)= 


nxm 

NxM 


Equation  10 


P{G,)= 


NxM  -  nxm 


Equation  11 


Equation  12  and  Equation  13  are  the  bivariate  normal  probability  distribution 
functions  for  the  target  and  background  respectively  and  represent  the  probability  of 
observing  the  values  v,  and  v,  given  that  the  pixels  belong  to  a  specific  group.  For  a 
given  pixel,  the  same  spatial  relationship  is  used  to  compute  the  target  likelihood  as  was 
used  to  compute  the  distribution  functions.  Therefore,  if  the  pixel  and  its  pair  have  values 
near  that  of  the  target  mean,  the  target  likelihood  is  increased.  Further,  if  the  relationship 
between  the  pixel  pair  values  is  similar  to  that  defined  by  the  target  correlation 
coefficient,  the  target  likelihood  would  also  be  increased. 


Equation  12 


|GJ- 


I  lrtat\\-p,  ] 


Equation  13 
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Based  on  Bayes’  rule  and  the  fact  that  there  are  only  two  groups,  pixels  with  a 
target  likelihood  value  greater  than  50%  are  classified  as  belonging  to  G, .  Using  this 
rule,  each  pixel  in  the  MG  is  associated  with  one  of  the  two  classes. 

The  Bayesian  metric  (Puay)  is  the  average  success  of  correctly  classifying  target 
and  background  pixels.  The  success  of  correctly  classifying  target  pixels  is  the  ratio  of 
correctly  classified  target  pixels  within  the  TPG  (Nct)  to  the  total  number  of  pixels  in  the 
TPG  (Ntpg)-  The  success  of  correctly  classifying  background  pixels  is  the  ratio  of 
correctly  classified  background  pixels  in  the  BPG  (Ncb)  to  the  total  number  of  pixels  in 
the  BPG  (Nbpg)  as  shown  in  Equation  14. 

P  -^CT  Ntpq  +  ^BPG  Equation  14 


The  Bayesian  metric  has  a  value  of  1  when  all  target  pixels  and  background  pixels 
are  correctly  classified  and  has  a  value  of  0  when  all  pixels  are  incorrectly  classified.  If 
all  target  pixels  are  correctly  classified  and  all  background  pixels  are  incorrectly 
classified,  the  metric  has  a  value  of  ¥2. 

In  summary,  the  Bayesian  metric  is  a  means  for  quantifying  the  separability  of 
target  and  background  statistics.  There  are  several  parameters  in  the  metric  that  are 
configurable  such  as  the  distance  to  the  pixel  pair  (XOFF  and  YOFF)  and  the  means  of 
establishing  the  statistical  model  (e.g.  models  based  on  current  frame  or  recursively 
updated).  After  exercising  the  metric  against  a  large  data  set  the  configurable  parameters 
should  be  studied  in  order  to  provide  optimum  performance. 

3.2.2.4  Signal  to  Clutter  Measure 

The  Signal  to  Clutter  Measure  (SCM)^  is  a  metric  that  predicts  the  probability  of 
detecting  a  target  in  an  infrared  scene.  It  was  developed  by  Margaret  A.  Phillips  and 
Richard  F.  Sims  of  the  AMCOM  Research,  Development  and  Engineering  Center.  This 
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metric  takes  the  target  signature  information,  as  defined  during  the  ground-truth  process, 
and  performs  a  correlation  with  the  entire  image.  This  is  greatly  different  then  metrics 
that  compare  target  area  statistics  to  local  statistics  around  the  target  area.  The  process 
for  performing  the  correlation  is  very  similar  to  the  TCM  correlation  process.  It  uses  a 
snap-shot  of  the  target  area  on  a  given  frame,  and  performs  a  normalized  cross  correlation 
with  the  entire  scene.  Correlation  peaks  outside  the  target  area  are  examined.  If  there  are 
peaks  inside  the  clutter  area,  then  there  are  clutter  features  that  could  cause  an  ATD 
algorithm  to  get  confused  and  ntis-classify  a  portion  of  the  background  as  target.  If  there 
are  few  or  no  correlation  peaks  in  the  clutter  area,  then  the  target  signature  is  not 
correlated  to  the  clutter  and  there  should  be  a  higher  probability  of  a  successful  detection. 
The  metric  also  examines  the  variance  in  the  complete  correlation  surface.  If  there  is 
variation  in  the  correlation  surface,  this  too  would  lower  the  probability  of  a  successful 
detection. 

The  output  of  the  SCM  is  a  floating  point  number  between  0  and  1  which  is 
consistent  with  the  desired  output  since  it  is  normalized.  A  1.0  would  indicate  a  good 
probability  of  detecting  the  target  while  a  0.0  indicates  a  very  poor  probability  of  target 
detection. 

3.2.3  Tracker  Performance  Metric 

In  order  to  grade  tracker  and  AT  A  performance,  a  tracker  performance  metric 
(TPM)  is  used  that  compares  the  ground  truth  gate  to  the  gate  generated  by  the  seeker 
algorithms.  The  TPM  used  for  this  analysis  independently  compares  the  ground  truth 
gate  width  and  height  to  the  tracker  width  and  height.  To  perform  this  comparison,  a 
normal  distribution  is  generated  using  the  center  of  the  gate  as  the  mean,  and  the  gate  size 
as  the  sigma  of  the  distribution.  The  overlapping  area  of  the  two  distributions  is 
calculated.  Equation  15  describes  the  TPM  calculation  where  firk(x)  and  ftrk(y)  are  the 
normal  distribution  functions  for  the  track  gate  x  and  y  dimensions,  and  fgt(x)  and  fgt(y) 
are  the  normal  distributions  for  the  ground  truth  gate  x  and  y  dimensions. 
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Equation  15 


Since  the  area  under  each  curve  is  1.0,  perfect  overlap  would  result  in  a  value  of 
1 .0.  Any  mis-match  would  result  in  a  lower  performance  number.  A  normal  distribution 
is  used  to  weight  the  center  of  the  gates  stronger  than  the  gate  edges.  The  normal 
distribution  will  give  more  emphasis  on  the  location  of  the  center  of  the  gate  to  ground 
truth,  and  less  on  gate  size.  Shown  in  Figure  15  are  plots  of  two  hypothetical 
distributions.  The  track  gate  width  is  smaller  than  the  ground  truth  gate  width  but  the 
gate  height  is  very  similar.  The  TPM  is  used  to  grade  system  performance  that  will 
ultimately  be  compared  to  the  output  of  the  metric  tool.  This  information  will  be  input 
into  the  neural  network  and  used  to  draw  a  correlation  between  the  actual  performance  of 
the  tracking  system  and  the  metric  values. 


Figure  15.  Tracker  Performance  Metric 


Since  the  metrie  values  are  a  measure  of  how  well  a  tracking  system  will  perform 
given  a  certain  target  to  background  situation,  the  metric  values  should  have  some 
relationship  with  the  acmal  values  as  measured  by  the  TPM.  Or  potentially  a 
combination  of  metric  values  with  associated  costs  values  will  be  used  to  predict  system 
performance.  In  an  example  case,  the  track  correlation  metric  was  used  to  compare 
against  actual  tracker  performance  for  a  specific  infrared  sequence.  These  sequences  were 
processed  using  the  traek  correlation  metric  and  the  tracker  evaluated  using  the  TPM. 
The  track  correlation  metric  data  was  compared  to  the  actual  system  performance  to 
determine  if  there  truly  is  some  relationship  between  this  metric  and  actual  tracker 
performance.  Figure  16  shows  this  comparison.  It  is  not  expected  that  these  plots  would 
match  exactly,  but  you  can  see  a  good  correlation  between  the  actual  tracker  performance 
and  the  metric  output. 


Figure  16.  Track  Correlation  Metric  Comparison  to  Tracker  Performance 

Performance  on  the  autonomous  target  acquisition  algorithm  is  more 
straightforward.  The  output  of  the  ATA  is  either  an  X,  Y  aimpoint  somewhere  in  the 
scene,  or  an  estimated  gate  size  and  location.  In  the  case  where  the  ATA  calculates  a 
gate  size,  the  TPM  mentioned  above  will  be  used  to  ealculate  the  performance  of  the 
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system.  A  metric  value  of  1.0  indicates  perfect  gate  placement  relative  to  ground  truth 
while  a  metric  value  of  0.0  indicates  no  overlap  of  the  normal  distributions.  If  the  ATA 
outputs  an  aimpoint  only,  then  the  performance  will  be  limited  to  a  1  or  zero,  depending 
on  whether  the  ATA  placed  the  aimpoint  successfully  inside  the  ground  truth  gates  or 
not. 


3.3  Signature  Metric  Software  Tool 

The  product  developed  under  this  effort  is  the  Metric  Analysis  Tool.  It  reads 
standard  image  sequence  formats,  as  well  as  raw  and/or  binary  formats.  It  is  written  in 
C++,  and  a  standard  metric  object  framework  was  established  to  facilitate  the  addition  of 
metric  routines  in  a  plug-and-play  type  environment.  It  has  the  ability  to  calculate  the 
metrics  and  output  results  in  various  forms  including  plots  to  the  screen  and  in  ASCII 
output  files.  In  Phase  II,  it  will  interface  with  the  performance  prediction  code  developed 
under  that  effort.  The  metric  tool  has  VCR -type  commands  such  as  play,  frame  step, 
rewind,  and  stop,  to  allow  viewing  of  the  image  sequences  as  they  are  processed.  The  tool 
contains  the  ability  to  select  a  large  number  of  image  sequence  to  process,  and  be  able  to 
select  the  metrics  that  will  be  executed.  It  can  also  be  run  from  the  command  line  with 
no  GUI  interaction  for  overnight  or  batch  processing.  It  was  developed  using  platform- 
independent  libraries  compatible  with  Windows,  Linux,  and  SGI  platforms. 

The  Metric  Software  Framework  will  provide  an  open-source,  modular,  scalable 
architecture  for  software  development  of  additional  metrics  and  interfaces  to  performance 
metric  algorithms. 

The  GUI  version  presents  the  user  with  an  interface  allowing  for  the  creation, 
editing,  and  execution  of  image  sequences  analysis.  The  image  sequences  and  the 
metrics  to  be  calculated  on  these  sequences  are  presented  to  the  user  in  a  list  view  format. 
There  are  also  three  data  view  windows  contained  in  the  main  program  window.  These 
contain  a  VCR  type  viewer  to  enable  viewing  of  the  image  sequence,  a  window 
displaying  the  metric  values  calculated  for  each  frame  in  the  sequence,  and  a  graphical 
representation  of  the  metrics  calculated.  There  are  two  dockable  sub  windows  involved 
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in  the  presentation  of  the  analysis  configuration,  the  Project  View  and  the  Property  View. 
An  example  of  the  GUI  can  be  seen  in  Figure 
17. 
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Figure  17.  Image  Metric  GUI 


The  Project  View  window  shows  a  hierarchical  list  containing  the  image  sequence 
file  names;  these  are  contained  under  the  parent  item  named  “Analysis”.  The  second 
parent  entry  in  this  list,  “Metrics”,  contains  the  names  of  the  metrics  to  be  performed. 
Image  sequences  and  metrics  can  be  added  and  removed  from  the  list  view 
independently.  Clicking  on  an  individual  entry  in  the  Analysis  list  will  cause  the  detail  of 
this  image  sequence  to  be  displayed  in  the  Property  View  Window.  This  action  will  also 
display  the  image  sequence,  if  it  exists,  in  the  VCR  window.  If  the  analysis  has  been  run 
previously  and  the  output  file  exists,  the  metric  data  will  be  presented  as  well. 
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The  Property  View  window  displays  the  individual  fields  describing  the  size  and 
location  of  the  image  sequence  file.  Also  presented  here  are  the  required  video  file 
decoder,  date  of  the  run,  output  file  name,  output  format,  ground  truth  filename  and 
ground  truth  file  format. 

A  menu  bar  in  the  main  program  window  provides  options  for  adding  metrics  to 
be  calculated,  the  sequence  to  process,  or  a  batch  mode  wherein  the  user  specifies  a 
directory  and  all  of  the  sequence  files  in  the  directory  will  be  added  to  the  analysis.  This 
batch  mode  also  fills  in  the  required  information  pertaining  to  the  sequence,  such  as 
ground  truth  file,  image  size  etc. 

The  GUI  also  provides  feedback  to  the  user  during  analysis  execution.  This 
feedback  is  provided  in  the  form  of  a  progress  bar  showing  percent  completion  of  the 
analysis  for  the  loaded  file.  If  the  file  contains  multiple  sequences,  a  new  bar  is  displayed 
as  each  sequence  calculation  is  performed.  The  GUI  is  multi  threaded,  so  the  interface 
remains  usable  while  the  calculations  are  being  performed. 

The  console  version  of  the  analysis  program  is  a  separate  application.  This 
version  of  the  tool  requires  the  parameters  to  be  specified  either  on  the  command  line 
individually,  or  alternatively,  a  configuration  file  generated  by  the  GUI  can  be  used.  This 
tool  provides  no  inspection  mechanism,  but  the  output  file  can  be  examined  in  the  GUI 
tool,  or  using  a  text  or  spreadsheet  application. 

The  configuration  file  generated  by  the  GUI  application  is  stored  in  the  Extensible 
Markup  Language  (XML).  Any  text  editor  may  be  used  to  edit  this  file  directly,  however 
this  is  not  recommended.  Also,  the  file  may  be  viewed  directly  using  a  web  browser. 
While  not  the  ideal  method  of  manipulating  the  file,  this  ability  can  be  useful  when 
troubleshooting  a  configuration  file. 

The  metric  tool  software  is  deliverable  as  either  a  binary  install  or  as  source  code 
with  configuration  files.  The  binary  install  is  currently  only  available  for  Microsoft 
Windows,  version  2000  and  above.  The  source  code  install  will  allow  the  tool  be  built 
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for  either  the  Windows  or  Unix  operating  systems.  New  metrics  may  be  built  and  used 
regardless  of  the  type  of  install  performed. 

Multi-platform  build  support  is  supplied  through  the  use  of  Trolltech’s  qmake 
utility.  This  utility,  along  with  the  configuration  files  it  uses,  will  be  supplied  with  the 
installation.  The  qmake  utility  generates  a  system  appropriate  make  file.  This  make  file 
can  then  be  used  to  build  the  executable  on  the  target  platform.  Detailed  build 
instructions  are  supplied  in  the  users  manual.  These  instructions  provide  information 
about  the  environment  variables  required  for  a  successful  compilation. 

The  GUI  interfaces  were  built  using  QT  by  Trolltech  Corporation.  This  is  a 
multi-platform  GUI  application  development  environment.  More  information  on  QT  can 
be  found  by  visiting  Trolltech’s  web  site  at  http:://www.trolltech.com.  This  web  site  also 
contains  detailed  documentation  on  the  qmake  utility. 

The  tool  also  makes  use  of  Invariant  Corporations  Itools  and  Codec  libraries. 
These  libraries  provide  support  for  the  Meta  Class  structure  used  in  the  metrics,  the 
image  sequence  decoding,  and  various  utility  data  structures  used,  Itools,  Codec  and  the 
metric  tool  in  general  also  make  liberal  use  of  the  Standard  Template  Library. 

The  source  code  install  includes  a  set  of  template  files  demonstrating  how  an  end 
user  can  extend  the  functionality  of  the  metric  tool  by  creating  their  own  metric 
calculations.  This  is  a  fairly  straightforward  process,  and  is  laid  out  in  a  cookbook  type 
description  in  the  metric  tool  user’s  guide. 

Several  environment  variables  are  required  for  proper  execution  of  the  tool. 
These  variables  and  their  values  are  described  in  the  users  manual. 

The  metric  tool  software  is  composed  of  two  major  components  and  several 
supporting  components.  The  two  main  components  are  the  GUI  object  and  the  Analysis 
object.  The  supporting  components  are  the  video  codecs,  the  analysis  codec,  and  the 
metrics. 
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The  GUI  component  encapsulates  all  of  the  operations  necessary  for  displaying 
the  GUI.  This  is  equivalent  to  the  “main”  program  in  the  console  application.  The  GUI 
also  contains  image  codecs,  the  viewer,  and  also  the  graph  object. 

The  Analysis  component  is  the  “heart”  of  the  metric  tool.  This  object  encapsulates 
all  of  the  items  necessary  to  provide  the  metric  objects  the  data  they  need  to  perform  their 
catculation(s).  The  Analysis  object  also  manages  calling  the  metric  objects  and  asking 
them  to  perform  their  calculations.  The  Analysis  object  contains  a  list  of  sub  objects 
called  sequences.  These  sequences  contain  all  of  the  information  associated  with  the 
image  sequences.  The  required  codec,  the  frame  range,  the  ground  truth  file,  the  output 
file  for  the  metric  values,  and  the  various  formats  of  these  items  are  all  kept  within  the 
sequence.  The  Analysis  object  also  contains  a  list  of  the  metric  calculations  to  be 
performed. 

The  tool  has  been  designed  using  object-oriented  methodologies  and  implemented 
Each  of  the  objects  described  above  correspond  to  a  class  in  C-i-i-.  The  metrics  are 
implemented  by  taking  advantage  of  the  polymorphic  characteristics  of  C++  allowing  the 
addition  of  new  metrics  without  recompiling  the  entire  application.  The  software  design 
Unified  Modeling  Language  (UML)  object  interaction  diagram  can  be  seen  in  Figure  18. 
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Figure  18.  UML  Object  Interaction 


Development  was  performed  using  Microsoft  Visual  C++  .Net,  version  2002. 

3.4  Ground  Truth  Process 

To  process  the  image  sequences  using  the  identified  metrics,  ground-truth 
information  of  where  the  target  is  in  the  image  had  to  be  obtained  or  generated.  This  was 
accomplished  using  the  Tracker  Analysis  and  Groundtruth  Tool  (TAGtool)  .  The 
TAGtool  runs  under  Windows  and  is  a  graphical  software  package  (see  screen  capture 
shown  in  Figure  19)  that  was  developed  by  Dynetics  for  AM  COM  as  a  means  of  quickly 
and  accurately  ground-tiuthing  infrared  image  sequences.  To  further  reduce  the  time 
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required  to  process  sequences,  a  correlation  capability  was  used  that  allowed  the  user  to 
ground-truth  every  tenth  frame.  The  auto-correlator  used  the  user-generated  information 
to  determine  the  ground-truth  data  for  intermediate  frames.  All  image  sequences 
identified  under  this  effort  were  ground-truthed  with  this  process. 


Figure  19.  TAGtooI  Screen  Capture 

3.5  Methodology  for  Seeker  Performance  Predictions 

3.5.1  Performance  Prediction  Methodology  Problem 

Presently,  a  comprehensive  framework  for  quantitative  analysis  of  missile  seeker 
designs  does  not  exist.  Although  parameters/metrics  such  as  3D  Noise  Statistics  have 
been  used  to  make  relative  comparisons  at  the  component  and  system  level,  the  utility  of 
existing  metrics  are  often  system  specific  and  depend  on  the  operational  functionality  of 
the  sensor. 
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Classic  performance  predictions  by  FLIR  92  and  NVTHERM  have  started  with 
empirical  information  such  as  Johnson’s  criteria  from  1950’s  and  examined  system 
performance  as  a  function  of  system  design  and  degradations  including  blur  and  noise. 
These  Man-in-the-Loop  (MITL)  performance  models  have  been  continuously  modified 
and  updated  to  reflect  state-of-the-art  in  sensors  coupled  with  human  operators.  The 
maturity  of  the  current  NVTHERM  models  allows  for  relative  comparisons  of  MITL 
systems  with  range  errors  of  ±20%  for  probabilities  of  detection,  recognition  and 
identification. 

For  Tracking  and  Target  Acquisition  Tasks,  characterizing  seeker  performance 
based  on  system  design  and  degradations  is  difficult  at  best.  Algorithms  used  for  these 
tasks  often  use  different  paradigms  and  information  to  process  incoming  images.  As  a 
result,  particular  image  metrics  are  often  not  indicative  of  relative  or  absolute  system 
performance  for  particular  task  such  as  Target  Acquisition.  The  goal  becomes  to  develop 
a  process  for  finding  and  combing  metrics  capable  of  predicting  task  performance.  In 
light  of  various  features  and  image  processing  techniques  available  to  accomplish  a  given 
task,  useful  metrics  may  vary  by  task  and  seeker  design. 

3.5.2  Performance  Prediction  Solution 

In  order  for  a  metric  or  combination  of  metrics  to  be  capable  of  estimating  task 
performance,  the  metrics  must  be  mapped  to  some  performance  measure.  In  the  case  of  a 
single  metric  (say  Delta  T),  standard  curve  fitting  might  be  used  to  map  a  particular 
metric  to  a  probability  of  detection  performance.  However,  a  combination  of  metrics  may 
be  more  robust  in  estimating  the  actual  performance.  The  key  is  having  criteria  for 
mapping.  For  a  tracking  task,  the  performance  might  be  measured  by  the  overlap  of  a 
target  ground  truth  box  and  a  system  track  gate.  The  desired  end  result  would  yield 
performance  estimation  for  a  particular  scenario  given  a  particular  system  and/or  a  set  of 
degradations. 

A  classical  Neural  Network  approach  provides  a  good  framework  for  determining 
a  mapping  from  a  set  of  metrics  to  a  desired  performance  measure.  Input  to  the  network 
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may  be  a  set  of  metrics  and/or  features  available  to  the  system  evaluator.  A  mapping  is 
generated  based  on  the  performance  measure  used  to  grade  a  particular  seeker  task. 

3.5.3  Development  of  Basic  Neural  Network  Approach 

To  begin,  a  weighted  sum  of  the  image  metrics  might  be  used  to  produce  an 
output  as  described  by  Equation  16  and  visualized  by  Figure  20. 

=  Equation  16 


Figure  20.  Linear  Spatial  Filter 

Here  the  mapping  between  a  set  of  input  vectors  (image  metrics)  and  desired 
outputs  (performance  measure)  is  known  empirically  for  selected  data.  The  task  is  to 
determine  the  weight  vectors  that  map  input  vectors  to  appropriate  output. 

An  error  signal  may  be  defined  as  the  difference  between  the  weighted  sum  y  and 
a  desired  output  d  generated  from  a  specific  performance  measure  (i.e.  Track  Quality). 

^  =  d  -y  Equation  17 

Minimizing  the  cost  function  defined  in  Equation  18  provides  a  basis  for 
minimizing  the  mapping  error  between  a  set  of  image  metrics  and  a  performance  measure 
(Track  Quality).  Here  E  is  used  to  represent  the  statistical  Expectation  operator. 
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Equation  18 


J  =-E[e^] 

2 

Minimizing  Equation  18  with  respect  to  the  weights  in  Equation  16  leads  to  the 
well  known  Weiner-Hopf  equations  for  determining  the  weights  Wj.  The  Method  of 
Steepest  Descent  is  often  implemented  rather  than  solving  the  Weiner-Hopf  equations 
directly  for  the  unknown  weight  vector. 

WjE[XjX^  ]  =  E[dx^  ]  ,  k  =  1,2,...,  p  Equation  19 


Weiner-Hopf  Equations 

The  Method  of  Steepest  Descent  determines  weights  via  an  iterative  scheme 
illustrated  in  Equation  20  until  changes  are  no  longer  significant.  Here,  n+1  represents  the 
updated  value  of  a  particular  weight  and  n  represents  the  previous  iteration. 


(n  + 1)  =  (n)  +  T] 


E[dk,]-f^WjE[XjX,] 


;=i 


k  =  \,2,...,p 


Equation  20 


However,  the  Expectation  operator  E  in  equations  17  and  18  indicates  the 
statistical  autocorrelation  and  cross-correlation  between  the  input  vectors  and  the  desired 
output  is  required.  In  many  cases  these  correlation  functions  are  not  known.  The  least- 
mean-square  (LMS)  algorithm  addresses  this  limitation  by  implementing  instantaneous 
estimates  of  the  correlation  functions.  The  net  result  leads  to  a  modification  of  Equation 
20  as  follows. 


Hi.(/i-l-l)  =  H\.(M)-i-;7[j(n)- >’(«)]  ,  k  =  l,2 . p 

where. 


yin)  =  Y,w(n)jX{n)j 

j 


Equation  21 
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Again,  this  is  an  iterative  technique  that  continues  until  changes  in  the  weight 
vector  are  no  longer  significant. 

So  far  these  approaches  are  applicable  to  linear  filtering  problems.  For  potential 
non-linear  mappings  of  the  image  metrics  to  a  performance  measure,  the  LMS  algorithm 
may  be  generalized  via  the  backpropagation  algorithm  for  Multi-layer  Perceptrons. 
Figure  21  illustrates  the  general  architecture  of  a  neural  network  with  a  single  hidden 
layer. 


Inpxit  Layer 


H,dd«Uyff  Output  Uy=r 


Figure  21.  General  Multi-Layer  Perceptron  Architecture 
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Figure  22.  Signal-Flow  Highlights  of  Output  Neuron  j. 

The  following  relationships  are  illustrated  by  the  signal  flow  graph  in  Figure  22 
for  a  neuron  in  the  output  layer.  Development  of  the  back  propagation  algorithm  begins 
by  defining  an  error  signal  for  any  neuron  in  a  similar  manner  as  for  the  LMS  algorithm. 

e  ■  (n)  =  dj  -  y'j  («)  Equation  22 

where 

ej(n)  =  error  signal  at  neuron  j; 
yi(n)  =  observed  output  at  neuron  j; 
dj  (n)  =  desired  out  put  at  neuron  j; 
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n  =  current  input  pattern 


The  cost  function  is  now  defined  as 
7=l£[e^.(n)^] 


Equation  23 


and  ej  represents  the  instantaneous  error  received  after  each  input. 

The  total  input  at  neuron  j  is  expressed  as 

^'j  ~  S  Equation  24 


The  output  at  neuron  j  is  a  function  of  Vj(n), 

yj  -  ^  )  Equation  25 


where  (p(  x  )  is  the  activation  function  of  the  neuron. 

Applying  a  similar  methodology  as  in  the  LMS  approach,  the  weight  updates  are 
proportional  to  the  instantaneous  gradient. 


dJ  dJ  dej(n)  dyj(n)  dv  (n) 

- —  =  T - T - T - T - =  -e-(n)(p  [v,.(w)]v,(/!) 

dwj.  dej(n)  dyj(n)  dvj(n)  dWj^in) 


As  before,  the  update  rule  is 


Wj.in  +  l)  =  Wj.{n)  +  AWj. 


where 


Equation  26 


Equation  27 
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The  sigmoid  function  is  continuous  for  all  values  of  x  and  ensures  a  well  behaved 
derivative  for  the  preceeding  equations. 

6j(n)  is  defined  as  the  local  gradient.  If  neuron  j  is  in  a  hidden  layer,  calculation  of 
ej(n)  is  not  straightforward.  From  Equation  25, 

_  dJ (n)  j(f^)  _  dJ(n)  Equation  29 

dej(n)dyjin)  dyj(n) 

For  clarity,  J  is  defined  at  the  output  layer  as 

Equation  30 


with  the  subscript  k  denoting  an  output  layer  as  opposed  to  the  input  layer  j.  From 
Equation  28,  the  goal  is  to  determine  the  gradient  of  the  cost  function  with  respect  to  the 
hidden  neuron  yj . 
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Equation  31 


Be,(n)  3g,(/2)  dv.jn) 

dyj(n)  T  *  dv^(n)dyj(n) 

where 


Ck  =  dk  -  (p[vk(n)] 


and 


Vk(n)  =  I  Wkj(n)  yj(n). 


Simplifying,  the  gradient  becomes 


Substituting  back  into  Equation  31, 


Awji  =  q5j(n)  yi(n) 


where 


6i(n)  =  (p'[vj(n)]  Z  5k(n)  Wkj(n) 
and 

Sk(n)  =  I  [ek  (n)]  (p'[Vk(n)]. 


Equation  32 


Equation  33 


Synaptic  updates  depend  on  whether  the  neuron  is  in  the  output  layer  or  a  hidden 
layer.  Output  neurons  use  Equation  31,  which  is  similar  to  the  update  rule  used  for  the 
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LMS  approach.  Hidden  neurons  use  Equation  32  where  the  local  gradient  depends  on 
synaptic  activity  in  the  output  layer  as  well  as  synaptic  activity  for  the  hidden  neuron. 

The  end  result  is  an  extension  of  the  update  rule  for  the  LMS  algorithm.  For  LMS, 
weighted  inputs  are  summed  at  a  single  node  or  neuron.  For  the  Multi-Layer  perceptron. 
the  error  is  ‘propagated  back’  through  the  network  to  provide  instantaneous  estimates  of 
correlation  functions.  The  increased  computation  complexity  provides  robust 
performance  for  mappings  that  are  not  linear. 

3.6  Validation  Plan 

This  section  presents  the  requirements  and  process  for  validation  of  the  Metric 
Detection/Track  Prediction  model  (MDTP).  MDTP  is  a  generic  analytical  IR  detection 
and  track  performance  prediction  model  described  in  this  report  and  proposed  as  part  of 
the  Phase  11  follow-on.  MDTP  validation  will  be  accomplished  by  comparing  the 
model's  metric  based  detection  and  track  predictions  to  field  test  results  utilizing  tactical 
and  generic  tracking  algorithms  and  validated  analytical  detection  models.  A 
comprehensive  set  of  IR  imagery,  previously  collected,  has  been  identified  under  this 
effort  that  encompasses  various  sensor/seeker  systems  engaging  an  array  of  targets  under 
various  environmental  conditions.  This  section  describes  the  methodology  for  utilizing 
this  data  set  and  outlines  the  necessary  steps  for  completion  of  the  validation  process. 

The  result  of  the  validation  process  will  be  a  metric  based  detection  and  track 
prediction  model  with  supporting  documentation,  which  can  be  confidently  used  as  a 
tool  for  prediction  of  infrared  (IR)  seeker/sensor  system  detection  and  track  performance 
for  a  variety  of  one-on-one  engagement  scenarios.  This  process  should  provide  insight 
into  the  validation  process  and  trade-offs  associated  with  model  fidelity  versus 
complexity  for  the  test  scenarios  under  study. 

3.6.1  MDTP  Metric  Tool 

MDTP  will  utilize  the  collection  of  metrics  identified  in  this  report.  It  will  use  the 
metric  software  tool  developed  under  this  effort  to  calculate  the  metric  values  on  new 
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infrared  sequences.  The  flexibility  of  this  framework  allows  the  ability  to  quickly  add 
new  metrics  if  deemed  necessary  under  the  future  effort.  The  neural  network  algorithms 
and  performance  prediction  algorithms  will  be  coded  and  integrated  into  the  current 
framework.  The  tool  will  maintain  the  ability  of  running  in  the  GUI  or  in  a  non-GUI 
batch  configuration.  The  GUI  will  have  an  experiment  planner  that  allows  the  user  to 
select  a  large  set  of  images  for  overnight  or  batch  processing,  and  will  continue  to  use 
platform-independent  libraries  compatible  with  Windows,  Linux,  and  SGI  platforms. 

3.6.2  MDTP  Probability  of  Detection  Prediction  Methodology 

Developing  a  detection  prediction  model  requires  output  from  a  representative 
algorithm,  the  metric  calculations,  and  ground-truth  information.  For  training  purposes, 
the  desired  output  of  the  network  is  set  according  to  algorithm  performance.  For  each  set 
of  metrics  generated  for  an  image,  a  performance  value  must  be  generated  for  driving  the 
desired  output  of  the  neural  network.  For  example,  the  output  of  the  network  may  consist 
of  nodes  representing  target  detections,  clutter  decisions,  and  false  alarm  predictions. 
Once  the  weights  of  the  network  are  determined,  the  output  values  of  each  node  are  used 
to  predict  how  the  detection  algorithm  wdl  perform  on  a  given  image.  Over  an  ensemble 
of  test  images,  the  neural  network  outputs  can  be  used  to  calculate  probability  of 
detection  and/or  false  alarm  rates. 

A  subset  of  images  from  the  data  sources  described  in  the  next  section  will  be 
used  to  train  the  neural  network.  Once  the  network  is  trained  for  a  desired  sensor,  metric 
inputs  to  the  network  can  be  used  to  predict  system  performance. 

3.6.3  MDTP  Probability  of  Track  Prediction  Methodology 

Tracker  performance  predictions  will  be  carried  out  in  a  similar  manner  as  for 
detection.  A  generic  tracker  algorithm  will  be  applied  to  various  image  sequences  with 
success  and  failure  determined  by  the  methodology  described  in  the  validation  section. 
Image  metrics  will  be  used  as  training  inputs  to  a  neural  network  while  the  success/failure 
results  will  provide  desired  network  outputs  and  feedback  for  determining  synaptic 
weights  between  each  neuron.  Once,  training  is  complete,  the  network  will  generate 
outputs  based  on  metric  input.  The  network  output  will  project  the  success  or  failure  of  a 
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tracker  for  a  particular  sequence.  Probability  of  Track  can  then  be  calculated  by 
observing  results  over  an  ensemble  of  metric  inputs  generated  by  a  set  of  image 
sequences. 

3.6.3. 1  Model  Validation  Methodology 

Validation  is  defined  in  DoD  Instruction  5000.61  as  “The  process  of  determining 
the  degree  to  which  a  model  is  an  accurate  representation  of  the  real  world  from  the 
perspective  of  the  intended  uses  of  the  model.”  The  Reconunended  Practices  Guide 
(RPG)  provided  by  the  Defense  Modeling  and  Simulation  Office  (DMSO)  describes  the 
essential  steps  for  validating  models  and  simulations  as  shown  in  Figure  23. 
Understanding  the  user’s  objective  and  characterizing  the  requirements  are  the  foundation 
of  the  validation  process  because  they  will  determine  the  accuracy  threshold  for  declaring 
the  results  valid.  This  threshold  will  be  determined  in  Phase  I,  and  the  validation  process 
will  be  executed  in  Phase  II.  The  results  of  tactical  tracking  algorithms  against  imaging 
data  collected  during  captive  flight  and  ground  testing  will  serve  as  the  available 
referents.  Simulation  may  be  used  to  fill  deficiencies  in  a  set  of  validation  data.  Signature 
metrics  calculated  from  the  same  set  of  image  data  will  be  used  to  predict  the 
performance  of  the  trackers.  The  accuracy  of  the  predictions  will  be  determined  by 
comparing  the  predicted  and  actual  tracker  performance.  If  the  accuracy  exhibits  the 
required  creditability  of  the  predictions,  they  will  be  deemed  valid. 
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Figure  23.  Essential  Steps  for  Validation  of  Models  and  Simulations 


MDTP  will  provide  an  approximation  to  the  stochastic  processes  of  target 
detection  and  tracking.  Therefore,  validation  of  MDTP  will  require  a  comparison 
between  the  deterministic  results  from  the  model  and  the  discrete  results  from  the  test. 
The  results  from  a  single  test  event  will  have  little  use  in  the  validation  process;  however, 
a  collection  of  test  results  can  be  used  to  calculate  the  probability  that  the  target  was 
detected  or  tracked  for  the  given  test  conditions  (i.e.,  m  detections  out  of  n  trials).  This 
calculated  test  probability  will  be  compared  to  the  predicted  performance  from  MDTP  for 
the  same  test  conditions.  Ideally,  a  collection  of  test  cases  in  which  all  conditions  remain 
constant  would  be  used  to  calculate  the  performance  probabilities;  however,  this  is  not 
practical.  Some  variation  in  the  test  conditions  of  the  test  cases  will  have  to  be  allowed. 
Restrictions  in  the  variability  of  the  test  conditions  for  a  set  of  test  results,  the  smaller  the 
test  set  will  be.  A  trade-off  between  test  condition  variation  and  test  set  size  (number  of 
test  results)  will  be  conducted  using  sensitivity  analysis.  The  allowed  variation  within  a 
test  set,  the  desired  number  of  results  within  a  test  set,  correlation  within  a  test  set  and 
other  validation  issues  are  addressed  in  this  section. 
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This  validation  methodology  is  described  in  detail  in  the  following  subsections. 
A  validation  notebook  will  be  used  to  document  the  steps  taken  and  used  for  the 
validation  final  report.  Figure  24  depicts  the  methodology  used  to  validate  MDTP. 


Figure  24.  Validation  Methodology' 


3.6.4  MDTP  Model  Validation 

Utilizing  a  portion  (approximately  half)  of  the  imagery  from  the  database,  a 
distinct  set  of  metrics  will  be  identified  that  will  provide  a  methodology  to  predict  the 
probability  of  detection  and  the  probability  of  track  for  a  given  a  scenario,  sensor 
characteristics,  and  environmental  conditions.  This  first  set  of  data  will  be  used  to 
develop  and  confirm  the  approach  for  developing  the  MDTP  model.  The  second  portion 
of  the  imagery  database  will  be  sequestered  for  use  during  the  validation  effort. 
Modifications  to  MDTP  may  be  made  prior  to  validation,  based  on  the  results  of  the 
metric  analysis.  Any  modifications  should  be  completed,  and  a  baseline  version  of 
MDTP  should  be  finalized  prior  to  beginning  the  validation  process. 


The  validation  effort  will  utilize  the  sequestered  data,  first  to  evaluate  the 
performance  of  generic  and  tactical  track  algorithms  for  comparison  to  the  MDTP  model. 


55 


This  effort  will  require  a  scoring,  or  assessment  for  each  tracker/image  sequence 
combination  for  the  success  or  non-success  of  the  event.  A  scoring  methodology  has  been 
previously  developed  for  tracker  assessment  and  approved  by  AMCOM.  This 
methodology  begins  by  attempting  a  track  at  the  farthest  range  through  endgame  (approx. 
250m).  Each  frame  is  scored  “in  Track”  if  an  overlap  condition  exists  between  the 
ground-truth  box  and  the  track  gate  box.  If  90%  of  the  frames  within  a  sequence  are 
tracked  then  the  track  event  can  be  declared  successful,  if  the  last  frame  of  data  is  not 
only  in  track,  but  the  center  of  the  track  gate  is  located  within  the  boundary  of  the 
ground-truth  box.  If  the  tactical  or  multi-algorithm  tracker  fails  then  the  slant  range  to 
target  is  decremented  by  250m  and  another  trial  is  attempted  using  the  same  imagery 
sequence.  This  methodology  can  have  the  affect  of  correlating  failed  cases,  especially  if 
the  cause  is  determined  to  be  clutter  near  the  endgame  of  the  sequence.  Therefore,  for 
single  algorithm  trackers  or  constant  slant  range  imagery  data  (  typically  generic)  an 
attempt  to  de-correlate  the  trial  data  is  accomplished  by  performing  track  attempts  on  200 
frame  segments  of  the  sequence,  resulting  in  each  trial  being  somewhat  independent  from 
the  result  of  the  previous  trial.  After  a  specific  tracker  has  been  run  on  a  section  of  the 
data  set,  bounded  by  specified  conditions  under  study,  the  overall  tracker  success  is 
determined.  This  success  will  then  be  compared  to  the  performance  of  the  MDTP  model 
prediction  for  the  same  data  set  to  determine  whether  the  model  correlates  well  with  the 
test  results. 

The  detection  performance  validation  of  autonomous  detection  algorithms,  as 
compared  to  MDTP,  will  be  handled  similarly  as  the  track  algorithms.  Selected  single 
frame  data  sets  will  be  used  for  both  MDTP  and  ATR  assessments  and  performance 
predictions  will  be  compared  for  validation. 

Validation  is  a  measure  of  comparison  between  the  MDTP  predicted  performance 
and  the  actual  performance  achieved  during  testing  of  various  track  and  ATD  algorithms 
and  perception  based  detection  models  for  the  imagery  database.  The  basic  validation 
process  consists  of  obtaining  data  from  pertinent  sources,  reducing  and  categorizing  it 
where  necessary,  and  compiling  it  in  a  table  format  for  comparison  purposes  on  a  mission 
set  basis.  These  mission  sets  will  contain  enough  data  points  to  be  statically  significant. 
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therefore  the  number  of  data  sets  and  track  algorithms  tested  should  be  optimally 
categorized  to  create  this  significance.  The  key  comparisons  to  be  made  are  Pd,  and  Pt  for 
a  given  set  of  measured  metrics. 

3.6.4.1  Sensitivity  Analysis 

A  sensitivity  analysis  will  be  used  to  reduce  the  dimensionality  of  the  validation 
process.  If  each  input  into  MDTP  were  treated  as  a  variable,  then  the  validation  would  be 
based  on  many  discrete  points  with  little  or  no  replication.  Sensitivity  analysis  can  be 
used  to  reduce  the  number  of  input  variables  by  making  several  of  them  constant 
throughout  the  validation  process.  This  creates  the  replication  required  to  achieve 
validation.  As  an  example,  it  is  anticipated  that  due  to  the  short  range  for  the  ER  sensor, 
the  atmospheric  transmission  will  remain  nearly  constant  over  a  given  slant  range. 

It  is  desirable  that  the  image  sequence  metrics  be  the  only  variations  between 
trials.  These  metrics  are  used  to  determine  the  Pd  and  Pt,  which  are  primarily  derived 
from  the  intensity  variations  between  target  and  clutter  background;  however,  if  a  limited 
number  of  significant  variables  are  introduced,  the  validation  still  can  be  performed. 

The  sensitivity  analysis  will  be  conducted  over  the  selected  data  set  as  part  of  the 
validation  process.  Clearly,  if  MDTP  is  relatively  insensitive  to  a  particular  parameter,  it 
is  unnecessary  to  measure  or  segment  that  parameter  to  a  high  degree  of  precision. 
Therefore,  the  sensitivity  analysis  can  be  used  to  refine  accuracy  requirements  for  the 
processing  of  the  imagery  database. 

To  conduct  the  sensitivity  analysis,  a  series  of  pre-designed  MDTP  cases  will  be 
run.  The  selection  of  these  cases  will  be  based  on  orthogonal  arrays  to  reduce  the  number 
of  runs  required.  Once  the  initial  MDTP  run  set  has  been  defined  and  the  most 
significant  variables  identified,  a  full  factorial  set  of  MDTP  cases  will  be  designed  and 
run  to  characterize  sensitivity  to  the  most  significant  variables. 
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3.6.4.2  CALCULATING  Pd  FROM  Test  Data 

The  test  data  will  be  analyzed  to  determine  the  associated  metrics,  as  well  as  the 
test  conditions.  In  the  simplest  cases,  the  Pd  or  Pt  will  be  determined  based  on  m 
detections/successful  tracks  out  of  n  trials: 


It  is  also  necessary  to  develop  an  MDTP  Pd  and  Pt  to  validate  against  the  imagery 
data.  The  conditions  and  metric  data  will  be  fed  into  MDTP  to  develop  these  values.  The 
scoring  for  the  test  data  will  be  on  a  per-look  or  per-track  basis,  since  the  process 
assumes  the  target  is  completely  within  the  field  of  view  of  the  sensor  for  each  image 
analyzed. 

3.6.5  Validation  Criteria 

3.6.5.1  Graphical  Analysis 

The  track  and  ATD  algorithms  will  be  plotted  alongside  the  MDTP  results.  This 
will  allow  for  direct  comparison  between  the  two  data  sets.  This  is  a  simplified  Turing 
test'*.  In  an  actual  Turing  test,  the  two  sets  of  data  would  be  displayed  with  no  distinctive 
markings.  If  an  expert  cannot  tell  the  difference  between  data  from  the  actual  algorithms 
and  from  the  simulation,  then  the  simulation  passes  the  validation  test.  For  this  analysis, 
plots  as  shown  in  Figure  25  will  be  examined  to  determine  if  MDTP  is  close  enough  to 
actual  test  results.  The  crucial  element  in  this  comparison  is  that  MDTP  is  not  required  to 
explicitly  reproduce  real-world  results.  It  is  only  necessary  for  MDTP  to  reproduce 
results  that  are  close  enough  that  decisions  made  based  on  these  results  would  be  the 
same  as  those  made  based  on  actual  algorithm  results. 
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RANGE  (m) 


Figure  25.  Graphical  Comparison  with  Confidence  Intervals 
3.6.6  Statistical  Analysis 

There  are  several  statistical  techniques  that  could  be  used  to  validate  MDTP.  The 
primary  statistical  technique  is  to  develop  a  confidence  interval  (Cl)  around  each 
performance  prediction.  This  technique  is  described  in  the  following  sections. 

3.6.6.1  Comparison  to  MDTP  Results 

An  initial  approach  to  comparing  test  results  to  MDTP  results  would  be  to  use 
hypothesis  testing.  The  hypothesis  to  be  tested  would  be: 


HO: 

Pd.-n: 

ST  “  Pd-MDTP 

HI: 

Pd-TH 

ST  ^  PdMDTP 

A  level  of  significance  would  be  chosen,  and  all  of  the  tests  would  be  run.  An 
equivalent  approach  is  to  use  CIs.  This  is  equivalent  because  at  the  same  level  of 
significance,  if  a  prediction  falls  outside  of  the  Cl,  then  it  also  fails  the  hypothesis  test. 
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CIs  have  the  advantage  of  allowing  a  more  understandable  form  of  presentation  (as 
depicted  in  Figure  25). 

The  Cl  represents  the  probability  that  the  true  performance  prediction  (Pd,  Pt)  is 
within  the  confidence  limits  with  a  certain  probability: 

P{Lower Limit  <  Pd  <  UpperLimit)=  1  —  a 

where  a  denotes  the  level  of  significance. 

The  MDTP  results  will  be  compared  to  the  calculated  CIs,  either  graphically  (as 
in  Figure  25)  or  in  tabular  form.  The  determination  of  model  validity  is  based  upon 
whether  the  MDTP  result  falls  within  the  CL  As  an  example.  Figure  26  shows  the 
possible  results  and  the  determination  of  whether  the  MDTP  result  is  valid  for  each  case. 
This  figure  also  shows  the  possible  sources  of  error.  Note  that  it  is  impossible  to  know 
whether  any  of  these  errors  have  occurred  without  knowing  the  tme  Pd.  However,  it  is 
possible  to  know  the  probability  of  committing  each  type  of  error.  The  method  of 
calculating  these  probabilities  is  given  in  the  following  sections. 

3.6.6.2  Type  I  Error 

In  the  equation  above,  the  level  of  significance,  a  is  the  probability  of  making  a 
Type  I  error.  A  Type  I  error  occurs  when  the  hypothesis  (HO)  is  true,  but  is  rejected  by 
the  sample.  Therefore  a  Type  I  error  occurs  when  the  Cl  should  include  the  true  Pd,  but 
does  not.  For  this  validation,  a  Type  I  error  is  depicted  in  Figure  26. 

It  is  expected  that  not  all  MDTP  results  will  fall  within  the  CIs.  Even  if  there 
were  exact  correspondence  between  MDTP  and  the  test  data,  it  would  be  expected  that  a 
certain  number  would  fall  outside  the  CIs.  This  number  is  based  on  the  probability  of 
making  a  Type  1  error.  The  probability  of  making  an  error  given  multiple,  Ud,  tests  (CIs), 
is  given  by: 

P  =l-(l-a)"''' 

error  V 
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where  the  level  of  significance,  a,  is  the  same  for  each  Cl.  Since  the  number  of 
errors  is  binomially  distributed,  the  expected  number  of  errors  in  na  tests  is  and. 
Therefore,  if  100  95%  CIs  are  developed,  (a=  0.05),  then  with  perfect  correspondence 
between  MDTP  and  the  real  tests,  5  of  the  MDTP  results  would  be  expected  to  be  outside 
of  the  CIs. 


CONFIDENCE  INTERVAL 


DETERMINATION  ERROR  TYPE 


MDTP 


MDTP  VALID 


NONE 


MDTP 


MDTPIINVALID 


NONE 


MDTP 


IMDTP  INVALID 


TYPE  I 


MDTP 


IMDTP  VALID 


TYPE  II 
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3.6.6.3  Cl  Development 


This  subsection  describes  how  to  develop  the  CIs  required  for  validation.  The  use 
of  the  standard  Cl  for  the  mean  is  not  accurate,  since  it  is  a  probability  (or  proportion) 
that  must  be  analyzed.  Therefore,  the  Cl  to  be  developed  is  a  Cl  around  a  proportion. 
For  large  sample  sizes,  this  can  be  approximated  using  the  normal  approximation  to  the 
binomial  distribution®: 


where  m  is  the  number  of  detections  in  n  trials,  and  XaJi  is  the  value  of  the 
standard  normal  distribution  that  has  a  cumulative  probability  of  cUi.  A  rule  of  thumb  for 
determining  a  sufficient  sample  size  is  given  by  Hicks®  as  nPd  t  4.  For  example,  if  the 
Pd  was  determined  to  be  0.5,  then  the  sample  size,  n,  should  be  at  least  8. 

The  exact  Cl  for  smaller  n  can  be  calculated  using  the  binomial  distribution. 
From  Beyer®,  given  that  the  lower  limit  of  the  Cl  is  represented  by  0^  and  the  upper  limit 
is  represented  by  0|„  the  lower  limit  is  calculated  such  that: 

x=x' 

where  a  is  the  level  of  significance,  x  is  the  number  of  detections,  and  n  is  the 
sample  size.  A  simplified  form  of  the  equation  above  is  given  by: 

x=0 

The  upper  limit  of  the  Cl,  0b,  is  calculated  such  that: 
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The  ideal  method  of  calculating  the  upper  and  lower  limits,  0a  and  6b,  would  be  to 
solve  the  above  equations  for  these  two  variables.  This,  however,  is  not  a  straightforward 
proposition.  Fortunately,  a  simpler  solution  will  produce  the  same  results.  This  simpler 
solution  is  to  iteratively  modify  0a  and  0b  until  the  resulting  cumulative  binomial 
probabilities  are  equal  to^S^ .  Although  there  are  tables  that  can  provide  these  solutions*’, 

with  the  advent  of  spreadsheets  with  powerful  statistical  analysis  functions,  more 
precision  is  available  through  the  use  of  an  Excel  spreadsheet.  Using  the  BINOMDIST 
function,  which  returns  cumulative  values  of  the  binomial  distribution,  a  spreadsheet  has 
been  developed  that  implements  Equations  7-10  and  7-11.  Using  the  Solver,  Excel  will 
iteratively  modify  values  in  cells  (specifically  0a  and  0b)  until  the  target  value  for  ^  has 
been  reached.  The  precision  can  be  set  to  almost  any  required  level. 


As  an  example  of  this  methodology,  consider  8  detections  in  30  trials  and  an 
alpha  of  0.05.  This  yields  a  Pd  of  0.266666667.  This  example  is  taken  directly  from 
Beyer^.  The  Cl  given  in  Beyer  (based  on  the  tables)  is: 

P(0.123<PJ<  0.459)  =  0.95 

while  the  interval  calculated  using  the  binomial  methodology  and  an  Excel 
spreadsheet  results  in: 

P(0.12279481  <  Pd  <  0.45889365)  =  0.95 

and  the  Cl  calculated  using  the  normal  approximation  results  in: 

P(0.108  <  Pd  <  0.158)  =  0.95 

The  Cl  calculation  clearly  provides  more  precision  than  is  possible  by  interpreting 
the  tables  in  Beyer  and  is  much  more  accurate  than  the  normal  approximation.  It  is 
relatively  straightforward  to  implement;  therefore,  it  will  be  used  to  calculate  the  CIs  for 
this  validation. 


63 


Note  that  solving  the  binomial  equations  works  only  for  m  out  of  n  successes 
while  m  is  in  the  range  1  to  n-1.  For  the  endpoints,  0  and  n,  a  slightly  different  version 
must  be  used.  For  0  out  of  n,  there  is  no  probability  associated  with  the  lower  limit; 
therefore,  only  an  upper  limit  can  be  calculated.  This  upper  limit  must  be  based  on 
oc  instead  of  0J2  to  ensure  a  consistent  Cl.  For  n  out  of  n,  the  lower  limit  must  be 
calculated  similarly  based  on  a 


It  is  instructive  to  examine  how  the  number  of  test  results  affects  the  width  of  the 
CIs.  Figure  27  depicts  the  CIs  calculated  using  the  binomial  methodology  for  the 
example  above.  The  bar  labeled  “30”  represents  the  Cl  for  8  detections  out  of  30  tests. 
The  “90”  bar  represents  24  detections  out  of  90  tests,  and  the  “300”  bar  represents  80 
detections  out  of  300  tests.  In  general,  the  more  test  cases  used  to  develop  the  proportion 
(Pd,  Pt),  the  smaller  the  CL 


Another  factor  that  affects  the  width  of  the  Cl  is  the  resultant  Pd  for  the  test  case. 
As  the  proportion  approaches  0.5,  the  Cl  becomes  wider.  The  smallest  CIs  occur  as  the 
proportion  approaches  low  or  high  values  (such  as  0.1  or  0.9).  This  effect  is  illustrated  in 
Figure  27.  This  figure  shows  the  width  of  the  Cl  determined  for  test  results  from  1/30  to 
29/30,  1/90  to  89/90,  and  1/300  to  299/300. 
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Both  the  number  of  tests,  and  the  tested  Pd  have  an  effect  on  the  Cl  width. 
Another  important  effect  is  on  the  Type  n  error,  as  will  be  shown  below. 

3.6.6.4  Type  II  Errors 

The  Cl  is  based  on  the  Type  I  error.  This  is  the  probability  of  rejecting  a  true 
hypothesis.  A  second  source  of  errors  that  will  be  considered  are  Type  E  errors.  The 
probability  of  occurrence  of  a  Type  E  error  is  represented  by  6.  6  is  the  probability  of 
accepting  a  false  hypothesis;  therefore,  it  is  the  probability  that  the  MDTP  result  is  within 
the  Cl,  but  should  be  outside  the  CL  Again,  this  error  is  depicted  in  Figure  26.  The  6 
error  can  be  calculated  based  on  how  large  a  discrepancy  is  acceptable  between  MDTP 
results  and  test  results. 


0.40  ~  - 30  TESTS 


0.05 


0.00  - -  - - - - 

0.0  0.2  0.4  0.6  0.8  1 .0 


Figure  28.  Cl  Width  versus  Pd 

Given  a  true  value  for  the  Pd,  it  is  possible  to  calculate  the  probability  of 
accepting  a  false  hypothesis  based  on  the  Cl  calculated  from  the  test  results.  Given  a 
lower  and  upper  limit  of  0^  and  0^,  the  probability  of  accepting  a  false  hypothesis,  6,  is 
given  by: 
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P  =  p[9,>Pd>0} 


where  Pd  is  the  true  probability  of  detection.  Calculation  of  6  is  then  made  by: 


p^p[pd<0,]-p[pd<e„] 

Assuming  a  normal  approximation  to  the  binomial  distribution,  a  Z  value  for  each 
of  the  above  probabilities  can  be  calculated  by: 


Z  = 


0-Pd 


lPd(l-Pd) 


Using  these  Z  values,  the  probabilities  to  use  can  be  determined  with  a  standard 
normal  distribution  table;  or  with  an  Excel  spreadsheet  function.  Thus,  6  can  be 
calculated. 

For  small  sample  sizes,  determination  of  6  must  be  based  on  the  binomial 
distribution.  All  of  the  equations  above  are  valid.  Determination  of  the  probabilities  in 
The  equation  for  P  above  (accepting  a  false  hypothesis)  is  somewhat  more  difficult  when 
the  binomial  distribution  must  be  used.  This  is  primarily  because  the  binomial 
distribution  is  discrete,  v/hereas  the  normal  distribution  is  continuous.  To  determine  6, 
Pd  is  used  as  the  mean  of  the  binomial  distribution.  The  number  of  trials,  n,  is  the  same 
as  the  number  of  tests  used  to  develop  the  Cl.  The  question  becomes:  what  is  the  number 
of  successes  in  the  cumulative  binomial  probability  equation?  The  number  of  successes 
is: 


Sa  =int(ean)  Sb  -int(ebn) 
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The  calculation  of  B  then  becomes: 


x=o 


^  ^  (i  -  Pd)-  - 1  (i  -  Pdy 

\x) 


This  is  a  straightforward  calculation  in  a  spreadsheet  using  the  previously 
described  functions.  Plotting  the  6  error  for  all  possible  values  of  pd  results  in  a  curve 
known  as  the  operating  characteristic  (OC)  curve.  Figure  29  shows  the  OC  curve 
calculated  for  the  example  problem  previously  described.  This  curve  shows  that  as  the 
true  Pd  gets  farther  from  the  test  Pd  used  to  develop  the  Cl,  the  probability  of  accepting 
that  the  true  Pd  is  within  the  Cl  decreases.  This  curve  yields  a  way  of  bounding  the 
accuracy  of  the  tested  Pd. 

As  can  be  seen  in  Figure  29  as  the  number  of  tests  increases,  the  range  of  Pd  with 
a  high  6  error  decreases.  It  is  possible  to  say  that  the  probability  of  accepting  the 
hypothesis  that  the  true  Pd  is  within  the  Cl  when  it  is  not  becomes  high  only  as  the  true 
Pd  is  very  close  to  the  tested  Pd  .  From  this  figure,  it  can  be  seen  that  if  it  is  desirable  to 
detect  a  shift  of  0.1  in  the  Pd,  then  the  probability  of  making  a  6  error  is  very  small, 
-0.01. 

For  this  validation,  a  minimum  detectable  difference  between  the  MDTP  Pd  and 
the  test  Pd  will  be  selected.  From  this  difference,  the  B  error  will  be  calculated  and 
identified  in  the  table  of  results  for  each  comparison. 
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Figure  29.  Effect  of  Number  of  Tests  on  OC  Curve 


3.6.6.5  DISCREPANCY  ANALYSIS 

It  is  impossible  to  predict  all  sources  of  error  that  would  cause  validation 
discrepancies.  This  subsection  briefly  introduces  several  potential  sources  of  error  as 
given  in  reference  4: 

1.  Errors  in  input  data, 

2.  Errors  in  procedure  or  use  of  the  model, 

3.  Errors  in  interpretation  of  results, 

4.  Errors  in  programming,  and 

5.  Errors  in  design  (algorithms). 


The  error  analysis  will  examine,  in  tiie  order  listed  above,  the  respective  errors  to 
determine  if  they  are  systemic,  random,  or  single  point.  It  is  important  to  note  that  usage 
and  programming  errors  will  be  examined  prior  to  algorithm  execution. 
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A  source  for  error  that  falls  under  item  2  (above)  is  the  fact  that  all  of  the 
measurements  are  taken  under  test  conditions,  and  that  there  are  multiple  looks  over  the 
same  vehicles  and  terrain.  A  modeling  assumption  is  that  each  of  the  looks  is 
uncorrelated  with  prior  looks.  This  assumption  will  be  examined  during  data  reduction  to 
test  its  validity. 

As  seen  in  Figure  24  there  is  a  feedback  loop  in  the  validation  process.  This 
represents  examination  of  errors  and  their  sources  and  modification  of  MDTP  inputs  or 
algorithms  to  more  closely  represent  the  test  results. 

The  support  the  tracker  performance  methodology  and  the  validation  plan,  a 
generic  tracker  must  to  identified  and  used  as  the  actual  tracker.  Several  portions  of  this 
report  allude  to  the  use  of  a  tracker,  but  are  left  ambiguous.  The  next  section  details  the 
actual  tracker  that  will  be  used  to  support  the  Phase  n  effort. 

4.  Tracker  Algorithm 

A  Modular  Framework  for  Algorithm  Development  and  Evaluation  (MFADE) 
and  the  Ground  Attack  Target  Engagement  (GATE)  algorithm  was  recently  developed 
the  Aviation  and  Missile  Research,  Development,  and  Engineering  Center  (AMRDEC), 
IR  Branch.  This  effort  was  performed  by  Dynetics  and  it  integrated  algorithms  for 
tracking  ground  targets.  Initially,  a  Hot  Spot,  Bayesian,  and  Feature-Based  Correlation 
(FBC)  algorithms  were  to  be  considered  for  inclusion  in  the  GATE  algorithm 

MFADE  can  be  run  from  its  Graphic  User  Interface  (GUI)  or  by  using  a 
command  line  version.  It  has  hooks  for  acquisition,  small-target,  mid-course,  and 
terminal  algorithms  and  can  support  multi-channel  algorithms  to  include  hooks  for  data 
fusion.  MFADE  is  implemented  in  a  modular  fashion  to  accommodate  growth  and 
expansion  in  the  future.  The  modular  design  also  can  support  insertion  of  real  image  data 
as  well  as  integration  with  simulations  that  include  seekers  and/or  scene  generators.  The 
GUI  is  focused  on  Microsoft  OS,  but  all  other  code  was  developed  to  ANSI  standards  to 
maximize  the  portability  to  other  computer  platforms.  MFADE  is  vmtten  in  C-i-i-  and 
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calls  many  subroutines  that  are  in  C.  Lastly,  there  are  hooks  for  interfacing  with  6-DOF 
models  or  external  data  sources  (e.g.  gimbal  data,  telemetry,  range  to  go,  etc.). 

The  primary  algorithm  within  MFADE  is  the  GATE  algorithm  which  consists  of  a 
combination  of  the  Anti-Median  Hot  Spot  (AMHS)  track  algorithm,  the  Anti-Median 
Geometric  Centroid  (AMGC)  track  algorithm,  a  re-centering  algorithm,  and  the  Feature 
Based  Correlation  (FBC)  track  algorithm.  The  primary  mode  for  the  algorithm  is  to  start 
in  the  AMHS  tracker.  The  imagery  is  filtered  using  an  Anti-Median  (AM)  filter  of  a  sub¬ 
image  around  the  target  area.  The  AM  filter  tends  to  enhance  hot  and  cold  spots  on  the 
target  while  suppressing  extended  bodies  such  as  roads,  poles,  trees,  and  so  on.  The 
GATE  algorithm  continues  in  the  AMHS  mode,  each  time  checking  to  see  if  there  is  a 
predominate  HS  (in  the  AM  filtered  image)  that  is  much  higher  than  the  surrounding 
background.  The  HS  inside  of  the  track  box  must  be  7  background  sigmas  above  the 
mean  background  level  (measured  in  a  background  box  surrounding  the  track  box).  If  the 
HS  intensity  ever  falls  below  this  level,  the  GATE  will  transition  to  a  re-centering 
algorithm  then  to  a  Sobel  AMGC  algorithm. 

The  re-centering  algorithm  is  accomplished  using  a  Sobel  Geometric  Centroid 
(GC)  routine.  First,  the  image  is  filtered  using  a  Sobel  routine.  The  Sobel  filter  is  a 
gradient  operator  that  enhances  the  rate  of  change  in  the  original  imagery,  which 
accentuates  the  edges  or  high  frequency  content  of  the  image.  The  idea  is  to  highlight  the 
target  edges  before  performing  a  GC  track  on  the  image.  This  is  repeated  for  25 
consecutive  images  in  an  attempt  to  walk  the  track  gate  onto  the  center  of  the  target  if  it 
had  previously  been  offset  because  of  AMHS  tracking  on  a  hot  spot.  On  each  frame,  the 
image  is  filtered  with  the  Sobel  filter,  then  the  top  12%  of  the  pixels  within  the  track  gate 
are  used  to  geometrically  center  the  new  track  box.  Note  that  this  re-centering  algorithm 
is  used  again  in  the  transition  to  FBC  from  either  the  AMHS  or  the  AMGC. 

The  Sobel  AMGC  algorithm  is  a  tracker  that  operates  on  an  image  that  is  first 
Sobel  filtered,  then  AM  filtered,  as  its  name  implies.  The  Sobel  AMGC  algorithm  has 
demonstrated  capability  to  track  targets  that  do  not  have  a  prominent  hot  spot  on  them, 
such  as  the  cold  side  (right  side)  of  a  T-72. 
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The  final  track  algorithm  that  is  invoked  is  the  FBC  algorithm.  A  transition  to  a 
re-centering  algorithm  is  performed  first  followed  by  the  FBC  algorithm.  The  transition 
occurs  when  the  track  box  has  more  than  750  pixels  on  target.  For  the  AMS  seeker  in 
narrow  field  of  view,  this  occurs  at  a  slant  range  of  approximately  1000  meters.  The  FBC 
algorithm  uses  a  reference  template  to  correlate  with  each  succeeding  image  to  locate  the 
target  and  center  the  track  box  on  the  target.  Additionally,  the  FBC  uses  a  feature 
extraction  and  a  scoremap  to  allow  processing  and  correlating  on  a  smaller  portion  of  the 
image  about  the  track  box  that  incorporates  only  features  that  are  persistent  ffame-to- 
frame.  This  results  in  fewer  calculations  and  completing  the  processing  more  quickly. 
The  FBC  algorithm  in  GATE  has  been  improved  from  its  predecessor  in  ISAT  for  better 
performance  in  endgame.  The  template  is  updated  more  often,  and  the  template  is 
magnified,  as  needed,  when  the  track  box  is  growing  at  high  rates,  such  as  during 
endgame. 

The  MFADE  and  GATE  were  developed  as  government  owned  and  operated 
source  code  and  algorithm.  All  results  can  be  openly  published.  And  since  it  is  a  robust 
tracker  and  should  represent  a  typical  tracker  used  for  missile  seeker  terminal  homing,  it 
will  be  used  as  the  system  tracker  in  the  methodology  implemented  in  the  Pending  Phase 
n  effort. 

5.  Conclusions 

In  conclusion,  all  requirements  for  the  completion  of  the  Phase  I SBIR  have  been 
met.  A  large  set  of  infrared  image  sequences  have  been  identified,  ground-truthed,  and 
processed  with  the  metrics.  Several  metrics  have  been  identified  and  developed  that  will 
support  future  analysis  and  the  Phase  11  effort.  A  modular  software  metric  tool  has  been 
developed  that  will  read  image  sequences  of  any  format,  and  process  the  sequences  with 
the  user  selected  metrics.  The  metric  tool  is  designed  for  ease  of  adding  additional  metric 
algorithms  as  they  become  available.  A  comprehensive  process  for  taking  the  metric 
outputs  and  comparing  them  to  actual  tracker  performance  for  the  training  of  a  neural 
network  process  has  been  defined  in  some  detail.  And  a  comprehensive  validation  plan 
to  prove  the  viability  of  the  performance  model,  and  then  ultimately  the  use  of  the  tool  by 


71 


the  community  has  been  identified.  The  actual  tracker  that  will  be  used  for  the  validation 
has  been  identified  and  has  been  approved  for  use.  And  the  proposed  phase  I  option  will 
allow  the  addition  of  MWIR  and  LWIR  image  sequences  and  ground-truthing  of  these 
sequences  for  use  in  the  Phase  H.  All  processes  are  defined  and  ready  to  take  this  effort 
to  the  next  level.  Clearly  the  feasibility  of  a  performance  prediction  capability  for  auto¬ 
tracker  and  ATD  systems  has  been  demonstrated  and  justifies  the  continuation  and  award 
of  the  Phase  II  effort. 
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1.  Introduction 


The  Invariant  Corporation  ImgMetrics  program  enables  the  user  to  perform 
various  metric  calculations  on  an  image  sequence.  The  metrics  are  calculated  on  a  “per 
frame”  basis,  resulting  in  a  set  of  metric  values  for  each  image  contained  in  the  given 
sequence. 

The  tool  is  extendable  by  allowing  for  end  user’s  to  create  their  own  metric 
calculations.  The  tool  is  also  versatile  in  that  the  set  of  metrics  calculated  for  each 
analysis  can  be  easily  changed  between  runs.  The  list  of  sequences  on  which  to  perform 
these  calculations  may  also  be  added  to  and  deleted  from  easily. 

The  tool  may  be  used  to  configure  the  analysis,  run  the  configuration  to  calculate 
the  metrics,  and  finally,  to  inspect  the  results  of  these  calculations.  The  tool  provides 
three  views  for  the  inspection  of  the  analysis  runs.  These  are  a  VCR  type  viewer  for  the 
image  sequence,  a  window  showing  the  exact  values  calculated  on  a  frame  by  frame 
basis,  and  a  graphical  representation  of  the  values  calculated  presented  over  the  range  of 
the  sequence.  The  graph  allows  for  the  selection  of  up  to  five  different  metrics  to  be 
graphed.  The  graphs  for  each  metric  are  distinguished  by  using  different  colors  for  the 
plot  lines. 

There  is  also  a  version  of  the  tool  which  provides  only  the  ability  to  run  the  metric 
calculations.  This  tool  runs  in  a  command  line  mode  and  produces  a  text  output  file. 

This  file  is  saved  in  CSV  format  and  may  be  viewed  using  a  text  editor  or  spreadsheet 
program,  such  as  Excel.  The  command  line  version  takes  either  all  of  the  parameters 
listed  individually  on  the  command  line  at  program  invocation,  or  alternatively,  will  read 
these  values  from  an  XML  file.  This  file  can  be  generated  by  the  GUI  version  of  the  tool, 
or  by  hand  using  any  text  editor  or  and  XML  editor.  The  format  of  the  XML  file  is 
provided  in  the  appendix. 


2.  Installation 

A  self  extracting  executable  install  file  named  ImgMetric_Setup.exe  is  provided 
to  install  the  ImgMetrics  tool.  This  install  file  has  been  created  using  Inno  Setup.  For 
more  information  on  Inno  Setup  please  see  http://www.irsoftware.org/isinfo.phE. 

The  install  file  contains  three  install  options.  These  options  are  “Source  and 
Configuration  files”,  “GUI  Version”,  and  “Console  Application”. 


2.1  Prerequisites 


There  are  a  few  pre-requisites  to  the  build  process.  One  of  these  is  the  QT 
development  environment.  QT  is  a  multi  platform  graphical  user  interface  programming 
environment  and  was  used  to  build  the  GUI  version  of  the  tool.  QT  is  a  product  of 
Trolltech  Corporation  and  more  information,  as  well  as  directions  on  obtaining  QT  can  be 
found  by  going  to  Trolltech’s  web  site  at  http://www.trolltech.com. 

The  GUI  version  also  uses  the  QWT  library.  The  QWT  library  is  used  to 
generate  the  graphical  representation  of  the  metric  values.  This  API  can  be  found  by 
going  to  this  website,  http:://  httn://Qwt.sourceforge.net/index.html.  The  website  will 
provide  further  information  on  the  use  of  the  QWT  library,  and  also  instructions  for 
downloading  and  installing  the  QWT  API. 

The  console  and  GUI  version  of  the  tool  also  use  the  Xerces  XML  API.  More 
information  about  Xerces  and  instructions  on  how  to  obtain  the  API  can  be  found  by 
going  to  this  website,  htqi://  http://xml.apache.org/. 

Each  section  contains  a  list  of  environment  variables  which  must  be  added  to  the 
target  system  and  then  to  that  system’s  path  environment  variable.  There  are  exceptions 
to  these  lists  if  the  target  system  is  already  using  Invariant’s  codec  and  ITools  libraries. 

If  this  is  the  case,  these  variables  will  already  exist  on  the  system  and  should  also  be  in 
the  path.  If  it  is  known  that  this  is  the  case  ignore  the  directions  for  setting  the  codecs 
and  ITools  environment  variables. 

Another  thing  to  be  careful  of,  if  the  libraries  already  exist  on  the  target  machine, 
is  version  incompatibility.  It  may  be  necessary  to  update  the  existing  versions  of  the 
libraries  with  the  new  ones  from  this  installation.  Simply  copy  the  dlls  from  their 
installed  locations  to  the  existing  location  on  the  target  machine.  This  may  cause 
imexpected  results  in  the  previously  existing  applications  dependant  upon  these  libraries. 


2.2  Source  and  Configuration  Files 

This  option  installs  all  of  the  files  necessary  to  build  the  tool  on  the  target 
machine. 

The  configuration  files  depend  on  the  existence  of  several  environment  variables. 
These  variables  are  used  by  die  configuration  scripts  to  generate  the  makefile  and  also  by 
the  software  as  it  runs. 

The  environment  variables  that  must  be  set  are  as  follows: 


Environment  Variable 

Description 

Metricsdir 

The  location  of  the  metric  dlls. 

Itools 

The  location  of  the  ITools  header  files. 

Codecs 

The  location  of  the  codecs  header  files. 

QTDIR 

The  location  of  the  QT  and  QWT  libraries,  as  well  as 
the  qmake  utility. 

ImgMetricsLibs 

The  location  of  the  AnalysisCodec  and  Metric  Base 
libraries. 

QMAKESPEC 

List  of  possible  values  can  be  foimd  by  looking  in  the 
QT/mkspecs  directory.  For  example  the  value  for 
Microsoft  Visual  C-H-.Net  would  be  win32-msvc.net 

XERCES 

The  location  of  the  Xerces  install. 

These  variables  must  also  be  added  to  the  path  environment  variable. 

When  these  pre-requisites  have  been  met,  the  system  specific  makefiles  can  be 
generated  using  the  Trolltech  utility  qmake. 

There  are  several  items  that  must  be  built  before  the  main  program,  ImgMetrics, 
is  built.  These  are  the  Analysis  Codec,  the  Metric  Base  object,  and  the  individual 
metrics. 

To  build  the  AnalysisCodec,  open  a  command  prompt  and  navigate  to  the 
AnalysisCodec  directory.  In  this  directory  type  the  command  “qmake’ .  This  will 
produce  the  system  appropriate  makefile  for  the  Analysis  Codec  library .  Once  the 
makefile  has  been  generated  build  the  library  according  to  your  compilers  instructions. 
For  example,  in  Microsoft  Visual  C++  .Net  version  2002,  you  would  now  type  “nmake”. 

To  build  the  Metric  Base  object,  navigate  to  the  MetricBase  directory  and  perform 
the  steps  listed  above. 

The  same  procedure  should  be  repeated  in  the  individual  metric  directories  to 
build  these  libraries.  The  metrics  are  located  in  subdirectories  under  the  MetricSources 
directory.  It  is  not  necessary  to  build  all  of  the  metrics.  Only  those  which  you  intend  to 
use  need  to  be  built. 

The  ImgMetrics  tool  can  now  be  built.  Perform  the  steps  listed  above  in  the 
ImgMetrics  directory  to  build  the  ImgMetrics  tool.  The  resulting  executable  will  be 
called  ImgMetrics.exe.  To  build  the  console  version  of  the  tool  perform  these  steps  in  the 
consoleApp  directory.  The  steps  preceding  the  makefile  generation  and  compilation  of 
the  main  tool  are  the  same  for  the  GUI  and  console  versions  of  the  tool.  With  the  one 


exception  being  that  the  console  application  does  not  need  either  the  QT  or  QWT 
libraries. 


2.3  GUI  Version 

This  option  will  install  a  ready  to  run  version  of  the  GUI  tool.  The  source  files 
will  not  be  included,  the  one  exception  being  the  template  outlining  the  creation  of  new 
metrics.  When  the  install  is  finished  the  following  environment  variables  must  be  added 
to  the  system  and  then  added  to  the  path  environment  variable.  These  variables  are: 

Metricsdir  —  The  location  of  the  metric  dlls. 

Itools  -  The  location  of  the  ITools  header  files. 

Codecs  -  The  location  of  the  codecs  header  files. 

QTDIR  -  The  location  of  the  QT  and  QWT  libraries,  as  well  as  the  qmake  utility. 

ImgMetricsLibs  -  The  location  of  the  AnalysisCodec  and  Metric  Base  libraries. 


2.4  Console  Version 

This  option  will  install  the  console  version  of  the  tool.  The  source  files  will  not 
be  included.  The  console  version  does  not  require  the  graphical  support  provided  by  the 
QT  and  QWT  libraries.  Therefore  the  environment  variables  for  these  libraries  need  not 
be  set  with  this  installation.  The  environment  variables  that  must  be  set  are  listed  below. 

metricsdir  -  The  location  of  the  metric  dlls, 
itools  -  The  location  of  the  ITools  header  files, 
codecs  -  The  location  of  the  codecs  header  files. 

ImgMetricsLibs  -  The  location  of  the  AnalysisCodec  and  Metric  Base  libraries. 


3.  Configuration 

The  only  configuration  issues  involved  are  the  setting  of  the  environment 
variables  listed  in  each  of  the  installation  sections  above.  The  individual  view  windows 
may  be  positioned  as  the  user  sees  fit. 


4.  Operation 

This  section  will  provide  detail  and  instructions  as  to  the  operation  of  the 
ImgMetrics  tool.  There  are  two  versions  of  the  metric  tool  available,  a  GUI  version  and 
console  application. 


4.1  Console  Application 


The  console  application  is  run  from  the  command  line  by  typing  the  command 
“consolelmgMetrics”.  The  program  expects  operational  parameters  to  be  supplied  on  the 
command  line.  There  are  two  ways  to  accomplish  this.  The  individual  parameters  can  be 
supplied,  or  the  name  of  an  XML  configuration  file  can  be  supplied.  The  individual 
parameters  expected  are  listed  below: 

start  -  Frame  to  begin  calculating. 

numFrames  -  Number  of  frames  over  which  to  iterate. 

vid  -  The  name  of  the  image  sequence  file. 

vdec  -  The  name  of  the  video  decoder  to  use  for  this  image  sequence, 
gtr  -  The  ground  truth  file  name, 
cal  -  The  calibration  file  name, 
output  -  The  output  file  name. 

metric  -  The  name  of  a  text  file  containing  the  metrics  to  be  calculated,  one  per  line. 


4.2  GUI  Application 

There  are  three  main  tasks  associated  with  the  GUI  version  of  the  ImgMetrics 
tool.  These  are  analysis  file  creation  and  editing,  executing  the  analysis,  and  inspecting 
the  execution  results. 


4.2.1  Analysis  File  Creation/Editing 


There  are  two  different  types  of  items  that  can  be  added  or  deleted  from  an 
analysis  file.  These  are  image  sequences  and  metrics.  These  items  appear  in  the 
ImgMetrics  GUI  along  the  left  hand  side  of  the  main  window  in  an  expandable  list.  The 
image  sequences  appear  under  the  main  heading  Analysis,  while  the  individual  metrics 
are  shown  under  the  main  heading  Metrics.  The  window  containing  this  list  will  be 
referred  to  throughout  the  rest  of  this  document  as  the  Project  View.  There  is  pother 
window  directly  below  the  project  view,  this  window  is  used  to  display  the  individual 
sequence  parameters  and  shall  be  referred  to  as  the  Property  View. 

To  add  image  sequences  to  the  analysis  select  the  Add  menu  on  the  menu  bar. 
This  is  a  drop  down  menu,  when  clicked  three  options  will  be  presented.  These  are  “Add 
Sequence”,  “Add  Sequences”,  and  “Add  Metric”.  We  will  discuss  the  "Add  Metric" 
option  a  little  later  on. 

If  “Add  Sequence”  is  selected  a  data  entry  form  will  be  presented.  The  form  can 
be  seen  in  figure  1  below. 
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Figure  1.  ImgMetrics  Add  Sequence  Form 


This  form  will  allow  the  entry  of  all  of  the  information  required  to  create  a  new 
sequence  on  which  to  perform  metric  calculation.  Where  file  names  are  required,  the 
data  can  either  be  typed  in  or  the  buttons  to  the  right  of  the  text  entry  fields  will  presented 
a  navigable  file  selection  dialog.  After  all  of  the  fields  have  been  filled  in  the  user  selects 
the  add  button.  This  will  add  the  new  sequence  to  the  list  of  sequences.  If  cancel  is 
selected,  the  data  entered  is  discarded  and  no  changes  are  made  to  the  file. 

If  “Add  Sequences”  is  selected  a  navigable  file  window  will  be  presented.  An 
example  of  this  can  be  seen  in  figure  2  below. 
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Figure  2.  Add  Sequences  File  Dialog 


The  file  dialog  will  allow  the  selection  of  multiple  files.  The  tool  will  fill  out  Ae 
rest  of  the  data  for  the  sequence  based  on  the  file  type  selected.  If  this  method  of  addition 
is  used  all  of  the  required  files  must  be  located  in  the  same  directory  as  the  image 
sequences.  Also,  the  calibration  file  must  be  named  the  same  as  the  parent  directory  with 
a  “.cal”  extension.  This  method  also  sets  the  Frame  Start  and  Frame  Count  parameters  to 
0.  The  program  will  determine  the  size  of  the  image  sequence  during  the  metric  run  and 
perform  the  calculations  over  the  entire  sequence. 

The  sequence  fields  may  be  edited  individually  at  any  time.  When  an  individual 
sequence  is  selected  from  the  analysis  list,  the  sequence  parameter  values  are  displayed  in 
the  Property  View  Window,  This  can  be  seen  in  figure  3. 


Figure  3.  Project  and  Property  Views 


When  the  individual  parameters  are  displayed  in  the  Property  View,  they  may  be 
double  clicked.  This  will  produce  an  edit  dialog  for  the  property  clicked  allowing  its 
value  to  be  changed.  The  change  can  be  accepted  by  clicking  on  the  Accept  button,  or 
the  changes  can  be  discarded  using  the  cancel  button.  An  example  of  the  edit  property 
dialog  is  shown  in  figure  4. 
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Figure  4.  Property  Edit  Dialog 


If  “Add  Metric”  is  selected  the  add  metrics  dialog  will  be  presented  as  shown  in 
figure  5. 
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Figure  5.  Add  Metric  Dialog 

This  option  will  allow  the  addition  of  metrics  to  be  calculated.  The  list  of 
available  metric  is  populated  by  the  metric  dlls  that  are  contained  in  the  directory 
indicated  by  the  METRICSDIR  environment  variable.  To  select  the  metrics  click  on  the 
metric  name  sin  the  available  list  and  hen  click  Load  Selected.  Load  ^1  will  move  all  of 
the  metrics  listed  in  the  Available  section  to  the  loaded  section.  Clicking  OK  will  add  the 
metric  to  the  analysis.  Cancel  will  discard  the  changes. 

Deletion  of  either  metric  or  sequences  can  be  accomplished  by  right  clicking  on 
an  item  in  the  Project  View.  This  will  present  a  pop  up  menu  with  the  delete  option. 


4.2.2  Executing  the  Analysis 

To  execute  the  analysis,  select  the  Run  Analysis  option  on  the  File  menu.  This 
option  will  calculate  the  metrics  contained  in  the  analysis  for  each  frame  of  every 
sequence  in  the  analysis.  The  results  can  be  viewed  at  any  time  after  the  run  is  complete. 

The  tool  will  show  a  progress  bar  indicating  the  percentage  complete  for  the 
current  sequence.  If  the  run  contains  more  than  one  sequence,  a  new  bar  is  presented  for 
each  sequence  as  it  runs. 

The  main  window  remains  active  during  the  calculations.  Sequences  on  which 
the  run  has  completed  may  be  viewed  and  the  metric  results  inspected.  If  the  analysis 
had  been  run  previously,  these  results  will  be  overwritten  by  the  new  run,  unless  the 
output  file  name  is  changed. 


4.2.3  Inspecting  the  Execution  Results 

After  an  analysis  run  is  complete,  or  having  loaded  a  previously  run  file,  the 
results  may  be  analyzed  using  the  ImgMetrics  tool.  The  tool  provides  three  view  of  the 

data  resulting  form  a  run.  These  views  are  presented  in  three  separate  windows.  These 

windows  will  be  called  the  Metric  Graph,  the  Metric  Data,  and  the  Sequence  View. 
Figure  6  shows  the  ImgMetrics  tool  displaying  these  views. 


To  begin  viewing  the  results  of  an  analysis  run,  select  the  desired  image  sequence 
fi’om  the  list  of  sequences  displayed  in  the  Project  View.  When  a  sequence  is  selected, 
the  sequence  parameters  will  be  displayed  in  the  Property  View,  as  discussed  above.  If 
the  image  sequence  exists  it  will  be  displayed  in  the  Sequence  View  window.  If  the 
analysis  has  already  been  run,  and  the  output  file  exists,  this  data  will  populate  the  Metric 
Data  window.  The  Graph  View  window  will  also  present  a  list  of  metrics  to  be  graphed. 

The  Metric  Data  view  shows  the  specific  values  calculated  for  each  metric  in  the 
analysis.  These  values  are  the  calculation  results  for  the  fi^ime  currently  displayed  in  the 
Sequence  View  window.  Navigation  through  these  values  is  accomplished  using  the 
VCR  type  control  buttons  in  the  Sequence  View  window. 

The  Sequence  View  window  begins  display  with  the  first  fi'ame  of  the  image 
sequence.  From  this  point  the  sequence  may  be  played,  stopped,  or  stepped  through  in 


forward  or  reverse.  There  is  also  a  scroll  bar  to  allow  the  sequence  to  be  scrolled 
through.  As  an  image  plays,  the  Metric  Data  window  will  refresh  with  the  currently 
displayed  frame’s  metric  values.  If  the  analysis  has  not  been  run,  but  the  sequence  exists, 
it  will  be  displayed.  However  no  metric  data  will  be  displayed  in  the  Metric  Data 
window  and  die  Graph  window  will  not  have  any  items  to  select  for  graphing. 

The  Graph  View  window  will  display  plots  of  the  metric  data  shown  over  the 
range  of  the  image  sequence.  There  is  a  pale  yellow  vertical  bar  displayed  in  the  plot 
area  indicating  the  location  of  the  image  frame  currently  displayed  in  the  Sequence  View 
window.  Up  to  five  metrics  may  be  concurrently  displayed  in  the  graph  area.  Each 
metric  will  have  a  different  colored  line  representing  the  values.  There  is  a  legend  at  the 
bottom  of  the  Graph  View  indicating  the  color  associations. 


4.2.4  Other  Operations 


The  ImgMetrics  tool  menu  bar  also  provides  for  other  standard  functions 
generally  expected  in  applications  today.  The  file  menu  provides  Save,  Save  As,  Exit, 
New  and  Open  operations.  There  is  also  a  help  menu  which  will  display  this  document. 


5.  New  Metric  Creation 

The  ImgMetrics  tool  metric  calculation  capabilities  are  extendable  through  the 
addition  of  user  defined  metric  calculations.  These  new  metric  may  be  built  upon 
currently  existing  metrics,  or  be  completely  original.  This  section  will  outline  the  process 
for  creating  new  metric  libraries  for  use  within  the  tool. 

The  different  variations  of  the  installation  all  provide  a  directory  called 
MetricTemplate.  This  ivill  be  a  subdirectory  of  the  Metric  Sources  directory.  This 
directory  initially  contains  three  files.  These  files  are  MetricTemplate.pro, 
MetricTemplate.cpp,  and  MetricTemplate.h.  These  files  contain  the  starting  point  for 
creating  your  own  metric  library. 

The  first  step  in  creating  your  own  metric  is  to  create  a  sub  directory  under  the 
MetricSources  directory.  Once  this  is  done,  copy  the  three  files  named  above  to  this 
directory.  Detailed  directions  on  foe  changes  to  be  made  to  each  of  foe  three  files  follow. 


5.1  MetricTemplate.pro 

This  file  is  to  be  used  with  foe  qmake  utility.  It  contains  all  of  foe  information 
qmake  needs  to  generate  foe  appropriate  makefile  for  the  current  operating  system.  To 
configure  this  file  for  use  with  your  metric  it  needs  to  be  renamed.  The  new  file  name 
should  match  the  name  of  foe  directory  just  created  in  the  step  above  with  the  .pro 
extension.  When  this  is  done,  edit  foe  file  and  change  all  occurrences  of 
“MetricTemplate”  to  foe  name  of  your  new  metric.  Typically  this  will  match  foe  name 


given  to  the  .pro  file.  For  example,  if  the  .pro  file  were  named  MyMetric.pro, 
“MetricTemplate”  would  be  changed  to  “MyMetric”.  Save  and  close  this  file. 

5.2  MetricTemplate.h 

This  file  should  also  be  renamed  accordingly.  Using  the  above  example,  it  would 
be  renamed  to  MyMetric.h.  Open  this  file  and  change  all  occurrences  of  MetricTemplate 
to  the  name  chosen  for  the  new  metric.  Save  and  close  this  file. 


5.3  MetricTempIate.cpp 

Following  the  pattern  illustrated  above,  rename  this  file  to  match  the  header  file. 
Continuing  to  use  the  MyMetric  example  this  file  is  renamed  to  MyMetric.cpp.  The 
occurrences  of  MetricTemplate  in  this  file  should  be  changed  to  match  the  name  used  in 
the  header  file.  When  this  is  complete  save  and  close  this  file.  We  are  now  ready  to  run 
qmake  to  create  the  makefile  for  the  new  metric. 

5.4  Finishing  Up 

Up  xmtil  this  point  the  directions  have  been  applicable  to  all  target  platforms. 

From  this  point  forward  the  directions  will  be  specific  to  a  platfom  running  Microsoft 
Windows  and  the  Microsoft  Visual  C-H-  .Net  version  2002  compiler. 

To  run  qmake  open  a  command  window  and  navigate  to  the  directory  containing 
the  new  metric  files.  In  the  new  directory  type  the  command  qmake  if  you  wish  to  build 
the  library  fi-om  the  command  line.  Alternatively  the  command  may  be  entered  as 
follows,  “qmake  -t  vcapp”,  this  will  create  a  project  file  with  a  “vcproj”  extension  for  the 
new  metric.  This  file  may  be  opened  using  the  C++  IDE. 

The  final  step  in  the  process  is  to  implement  the  printHeader  and  calculate 
methods  in  the  source  files.  The  printHeader  method  should  output  a  comma  separated 
list  of  the  metric  values  you  will  be  calculating.  There  should  be  no  new  line  contained 
in  the  list.  The  second  method  to  be  implemented  is  the  calculate  method.  This  is  where 
the  actual  calculation  of  the  metric  is  to  be  performed.  The  last  thing  calculate  should  do 
is  output  the  result  of  the  calculation,  followed  by  a  comma.  Again,  no  new  line  should 
be  inserted. 

When  the  changes  have  been  completed  build  the  library  by  running  nmake  from 
the  command  line,  or  the  library  can  be  built  inside  the  IDE  using  the  build  command. 
When  the  build  command  is  finished  the  new  library  will  be  placed  in  the  METRICDLLS 
directory  and  is  now  available  to  the  ImgMetrics  tool.  The  new  library  will  appear  in  the 
Available  Libraries  list  displayed  by  clicking  the  Add  Metrcis  menu  item  in  the 
ImgMetrics  GUI.  Good  Luck  and  Have  Fun! ! 


6.  Metric  Template  Source 


6.1  MetricTempIate.h 


//' 

// 

// 

// 

// 

$ 

// 

// 

// 

// 

Ih 


Infrared  Scene  Metrics  Program 

$workfi1e::  MetricTempIate.h 

$Revision::  1 

$Date::  11/12/03  10; 46a 

$Modtime::  11/12/03  10:45a 


//  MetricTemplate  is  the  "recipe"  for  creating  your  own  metrics  to  use 

with  the  metric  tool.  ,  .  n  .  ,  t  j 

//  All  of  the  "MetricTemplate"  keywords  should  be  replaced  with  your 

class  name. 


#include  "MetricBase.h" 
#include  <iostream> 


*  implementation  of  RectimgMetricBase  for  calculating  the  image 
statistics. 

*  calculates  the  scm. 

* 

*  @1.0 

*  ©scm.h 

V 

//  DLL  specifiers. . . 

#ifdef  STATSMETRIC_BUILDDLL 

#define  statsmetric_declspec _ declspecC  dll  export  ) 

#defi ne  STATSMETRIC_EXPIMP 

#elif  defined  statsmetric_DLL  ,  ^  ^ 

#define  statsmetric_declspec  declspecC  dll import  ) 

#define  statsmetric_expimp  extern 

#define  statsmetric_declspec 
#define  STATSMETRIC_EXPIMP 

#endi f 

class  MetricTemplate  :  public  RectimgMetricBase 

public:  .  -  1  * 

typedef  TimgMetri ccl ass<RectlmgMet ri cBase , Metn  cTempl ate> 

MetaClass;  .  ^  , 

static  imgMetricMetaClass*  Class; 

Metri CTempl ate() ; 

~Metri CTempl ate () {} ; 

/** 

*  calculate  the  metric  for  the  given  frame  using  the  GroundTruth 

*  provided.  The  result  is  written  to  the  file. 


*  ©param  Timage  The  image 

*  ©param  RectangularGroundTruth  The  groundtruth 

*  ©param  OStream  The  output  file  stream 
*/ 

void  printHeaderCstd:  :ostreain&) ; 

*  calculate  the  metric  for  the  given  frame  using  the  Groundtruth 

*  provided.  The  result  is  written  to  the  file. 

* 

*  ©param  Timage  The  image 

*  ©param  RectangularGroundTruth  The  groundtruth 

*  ©param  OStream  The  output  file  stream 


cal  cul  ateCTimage<f  loat>&,  Rectangul  arGroundTruth&,  std . .  ost  ream&) 

};//End  MetricTemplate 

//EOF 


6.2  MetricTemplate.cpp 


//  infrared  Scene  Metrics  Program 


//  Sworkfile::  MetricTemplate.cpp 
//  $Revision::  1 
//  $Date::  11/12/03  10:46a 
//  $Modtime::  11/12/03  10:45a 


//  This  is  the  "recipe"  for  creating  your  own  metrics  to  be  used  with 

the  metric  tool.  All  of  ,  .  .  .  ■  x 

//  the  "MetricTemplate"  keywords  should  be  replaced  with  the  name  of 

your  class.  The  printheader  ,  ■  u  j 

//  method  gets  called  once  at  object  creation  to  put  the  metric  header 
in  the  output  file.  The  calculate  ,  ,  .  .  . 

//  method  gets  called  for  each  frame  of  data  in  the  image  sequence. 

The  metric  calculation  code  goes  here.  .  n  .  .  ^  • 

//  private  methods  can  also  be  added  to  provide  modularity  in  metrics 
requiring  complex  or  detailed 
//  calculations. 


#include  "MetricTemplate. h" 

#i ncl ude  "vi deo/Vi deoDecoder . h 
#include  "cpputil/cstring.h" 

#i ncl ude  "cpputil/TSubArray.h" 

#i ncl ude  "cpputi 1 /TScal arAr ray . h 
#i ncl ude  "numeri cs/StatsAl go . h" 

#i ncl ude  "eo_i r/RectGroundTruth . h" 
#i ncl ude  "eo_i r/Cal i brati on . h" 


#i ncl ude  <iostream> 
#i ncl ude  <cstdlib> 

#i  ncl  ude  <algorithtn> 


#include  <stack> 


using  namespace  std; 
using  namespace  util; 

imgMetricMetaClass*  MetricTemplate: iClass  = 
MetricTemplate: :MetaClass: :instance() ; 

Metri cTempl ate : : Met ri cTempl ate () 

:  RectimgMetri cease  Q 

^  std::cout  «  ’’MetricTemplate  construct"  «  std::endl; 
}//  end  constructor - - - 


//  Print  the  headers  to  the  output  file 

void  MetricTemplate: iprintHeader (std: :ostream&  output) 


//  TODO:  Put  the  print  header  code  for  the  metric  here. 

output  «  "MetricTemplate  header  line  goes  here  «  std::end1;; 
}//End  of  printHeader  — - - 


//  calcualte  the  metrics.  _ 

void  MetricTemplate: : calculate CTimage<float>&  imageFrame, 

RectangularGroundTruthA  gt, 
std: :ostream&  output) 

//  TODO:  The  metric  calculation  code  goes  here.  ^  . 

std::cout  «  "MetricTemplate  example  code,  put  metric  calculation 
code  here."  «  std::endl; 


}//End  calculate 


//EOF 


6.3  Sample  XML  File 


<?xml  version="1.0"  encoding="UTF-8"?> 

<analysis> 

<sequences> 

<sequence  format="ams"  start_frame="l"  num_frames="159"> 

<path>C:\\SBIR_Data\\AMS_FLIGHT_10\\NightTest4a.ams</path> 

<calibration  format=”text "> 

<path>C:\\SBIR_Data\\AMS_FLIGHT_10\\AMS_Flight_10.cal</path> 

</calibration> 

<ground-truth  format=”amcom”> 

<path>C:\\SBIR_Data\\AMS_FLIGHT_10\\NightTest4a.gtr</path> 

</ground“truth> 

<output  format==”csv”  date="12-03-2003"> 

<path>C: \\users\\jnmont\\projects\\metrics\\AnalysisCodecTst\\j 

Stats . csv</path> 


</output> 

</sequence> 

</sequences> 

<metrics> 

<metric>VarianceMetric</metric> 

</metrics> 

</analysis> 


